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LEARNING STRUCTURAL DESCRIPTIONS FROM EXAMPLES* 



Abstract 



The research here described! centers on how s machine 
can recognize concepts and learn concepts to be recagnited*. 
Explanations are found in computer prograias thst build and 
manipulate abstract descriptions of scenes such as choSfr 
children construct from toy blocks. One pro&rani uses Sample 
scenes to crests models of simple conf ljiirat i cms like the 
three-brick arch. Another uses the resulting node Is in 
making identifications. Throughout fcmphasts is given to 
the importance of U4lng goed descriptions when exploring 
hov machines can cobe to perceive and understand the visual 
erivJ n>nmt'tit. 



*Thifl report reproduces a thesis of the same title submitted to 
the Departaent of Electrical \tn% inhering, HesBachusetts Institute 
of Technology , in partial tuIfiilMnt of the requirements for the 
degree of Doctor of Philosophy, January t Q 7Dr 
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1 Key Irfeas 
How do we ret Denize examples of various concents? 
How do we learn to make SiK h reconn 1t iens? 
Her can machine 5 He these t fit has? 
How IpSfi rt^ t an t Is earpful teaching? 

In this naper I describe 9 system that sheds sone llcht 
on these ouestions by demon strat inr hew a machine can he 
taught to see and learn nw visual concepts, It rorfe* i" the 
domain of three-dimensional structures made of hrlclLS, 
^etffles, and. other staple objects* 

Good descriptive netheds ar* of central Importance to 
this ^rk. This is eSenimstrat ed repeatedly In py system's 
facilities for scene description,, description ccrndri son , 
concept learning &nrt identification, 

It !s ny opinion that the framework fpr learning that 1 
describe suocjests a unity hetween Wrninn frm- examples, 
learning by imitation* and leamfnc by fcfllnr- told. This 
unity lies in the necessary ability to generate and 
manipulate good, abstract descriptions. 

I also grcrue the inpertance of noocl training sequences 
prepared by good teachers. I think it is reasonable to 
believe that neither machines nor children can be expected to 
learn nucn witnout thett* 

Equally important is the nation of the near itilss* Ey 



near miss I mean a sample 1n a training seouence nuite like 
the concept tq be learned but which differs frrm that concept 
In only a few sfonfffcant points at most. These near nisses 
prove to convey essential points much more directly than 
repetitive exposure to ordinary examples, 
1.1 Scene Description ard Comparison 

Much of the system to be described focuses on the 
problen of analyzing sc ere s consisting of the simple objects 
that one finds 1n a child's toy box. There »rt- two very 
simple examples of such scenes in fiiure 1*1. 

Froir such visual 1nao.es p the system builds a very ooarse 
description, (figure ~\*2) Structurinn the scene's description 
in terns of objects is already a certain comn ftmen t, for 
Structuring it in other terms 1s oossihle. In any case, 
analysis proceeds, inserting more dptall. (flour a 1-3} Ard 
finally there is the very fine detail ahout the surfaces, 
lines, vertexes, ard their relations. 

Such descriptions permit one to conpare and contrast 
scenes throuah programs that conpare and contrast 
descriptions, Of course, one hopes that the descriptions 
will be similar or dissimilar to the sane dearee that the 
scenes they represent seen similar or dissimilar to hunart 
intuition, Then with a oeneral plan for such Manipulations, 
there is further hope that the same machinery can be useful 
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in situations panging far from visual ones, niving the wort a 
cert*in generality* 

Certainly the flee es sary ma tchino pronrams must ha welT 
endowed with ability, for a rich description capability 
requires a matching program that can c ope vith and perform 
reasonably in an environment where many matches are possible, 
both good and bad. 

After two Scenes arc described and correspond in g carts 
related by the natch inn uronram^ differences in the 
descriptions must tie found, catecor 1 zee? t and themselves 
described, The program that does this must be able to 
examine the descriptions of fioure 1-3 with the help of a 
matchlnq program and deduce that the difference between the 
scenes is that there is a su, rp or ted -by relation 1n one case, 
while there is an in-frgnt-of relation in the other, Dut the 
faculty must be nut h more powerful than this sample example 
indicates in order to face more complex p^irs of scenes 
exhibiting the entire spectrum between the nearly identical 
and the compl etet y d If f erent , 
1.2 Concept Generation and Learn in ti 

To build a machine that can analyze line draining and 
build descriptions relevant to some comparison procedure 1s 
u&eful in itself. tint this is just a step toward the more 
ambitious goal of c^oatinn. a program that can learn to 
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recognize structures. I will describe 4 pr coram that can use 
samples of simple concepts to oenerate mndels* 

Figure- 1 -■* and the next few followinq 1t show a sequence 
of samples that enables the machine to learn what an arch is. 
First ft gets the general idea by studying the first sample 
in f inure 1-4. Then it 1 Earns refinements to Its orlnlnal 
conception by comparing its current Impression of v-'hat an 
arch 1s with successive samples* It Teams that the supports 
of an arch cannot touch from figure 1-5. It learns that it 
does not matter much what the top object Is from figure 1-6. 
And then from f inure 1-7 it learns the fact that for one 
object to do supported foy the others is a definite 
requ Iremen t, not just a cnincidonce carryini throurrh all of 
the samples. 

Such new concepts can in turn help in ma k inn other* wore 
complex abstractions, Thus the machine uses previous 
learning as an aid toward further learning and further 
analysis of the env fronm en t , As yet these procedures are 
clumsy, and the descriptions uneonf ortabl y restricted, but 
the result* are encouran^ng eneuah to suggest that these 
methods nay lead to Increasingly powerful performance. 
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1 ,3 Jdentlf Icatfon 

Identification requires additional proqrams that use the 
results of comparison programs* There are many problems and 
ma ny alternative method s Involved because identification can 
he done in a. variety of contexts. 

In one simple fern of IHen t ff lea t ion , the machfne 
compares- the description of some scene to be Identified with 
a repertoire of models, or stored concepts, Then at the very 
least there must be some method qf evaluating the comparisons 
between the unknown ancf the mo-dels so that some match can be 
d e f In ed as b* ?t » 

But many sophistications lie beyond this skeletal 
scheme. For one thing, the identification can be sensitive 
to context. In fiqure l-&> for example, one hidden object 1s 
n-.ore likely to be a hredoe than in the other case, although 
both hidden objects present exactly the s.a*ie line 
configuration* The Identification could be further 
prejudiced if the objective is to locate a particular type o-f 
object. Thus the hidden object in figure 1-9 should he 
tentatively Identified as a possible trapezoidal solid, 
rather than a wedoe, if trapezoidal solids are 1n demand. 
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1,4 Psychol og i ca 1 Modeling 

Simulation of human intelligence is not ft primary goal 
of this work. Vet for the most part I have designed proqrans 
that see the world in term* confirm Ino to human usacie and 
taste* These programs produce de scr fpt 1-ort-s that use notions 
such as left-of, on-top-of* behind, big, and part-of* 

There are several reasons fur this. One is that If a 
machine is to learn fro-^ a human teacher, then 1t is 
reasonable that the machine shoud understand and use the same 
relations that the human: does, ft t her vise there would he the 
sort of difference fn point of view that prevents 
inexperienced adult te-achrrs frnn interact Inn smoothly with 
sma 11 c h f Idren . 

Moreover, if the machine Is to understand its 
environment for any reason, then understanding it in the same 
terms humans do helps us humans to understand and iirnrove the 
machine's operation. Little is known about how human 
intelligence works, but 1t would be foolish tn ignore 
conjectures about human methods and abilities if those things 
can help machircs. Much has already been learned from 
programs that use what seem like human methods. There &r& 
already proqrans that prove mathematical theorem^ oIav good 
chess, work analogy problem!;, understand restricted fnrms of 
English, And more. Vet in contrast, little knowledge about 
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Intel 1 Igence has come from percentron work and ether 
approaches t& Intel 1 locnce that do not exploit the plannlnn 
and hierarchical orqa n fzat ion that is characteristic of hurran 

t houflht ,. 

Another reascn for designing nrnnrans that describe 
scenes in hu*ian terns is that human judnepent then serves as 
a standard. There will be no cont en tment with machines that 
only do *s well as huinans* But until machines becc-we better 
than humans at seeing, dninr as well is 3 t*e*iscnablp noal, 
and comparfnc the performance of the machine wftti that Pf the 
human is a convenient way to measure success- 
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2 Building oeser ipt inn s 
2 k ~\ The Uetwrk 

There arc many ways to store facts a Lout 3 scene. Ore 
simple format Is the unordflreti list: 

A f s n to p of D 

A 1 1 s a s f d e of A 

D is in front of C 
Such an arrangement is desperately inefficient because the 
whole Of KieiflOTy mi/st be searched to oathor all facts aoout 
some particular ccirponent of the scpha, \t is natural* 
therefore, to record facts In a mere structured va y to 
facilitate retrieval* 

In this connection, one hears such terns as lists, 
trees, rims, and nets, each of l/hith suggests a form of 
storane. In selecting one., attention must be nalri to several 
criteria* I have already mentions! the problem of rapid 
access, Thorn may also b-e a need to u'qe memory space 
efficiently, Ujt in the research nhas&, nerhans it Is flost 
important that the storage format be ^n sooe sense natural 
*1th respect to the information to be stored* This neans 
tPiat the tra nsf crmat ion from a situation to its 
representation should be sinnle*. not awkward* Single Hsts 
suffice for a trir. to tfop grocery store, vhile tree^like 
Charts frequently picture command hierarchies nr qenea 1 oqical 



lft 



h t s t o r i e s . 

But nany nflre complex situations require the net. A 
oood example is the description of the words in ft natural 
language. Each word is described easily In ter^s of 
relationships with other words which in turn are similarly 
described, T tie result is a dictionary in which each uor-rl my 
be thought of as a node which is re la t erf to other nodes 
throutfh the pointers that constitute its definition* 

Similarly the network seems to htva the appropriate 
blend of flexibility and elegance needed tc Heal 
stra fghtforwarl y with scenes. It is the natural format. 
Like words in a dictionary* each object is naturally thought 
of in terms of relationships to other objects and to 
descriptive concepts Hke large, rectanqul a r, and standing. 
In fiogre 2-1, for example, one has concepts such as OBJECT* 
ABC and OBJECT-DEF, These are represented d iatrrawmst ica 11 y 
as circles, {fioure 2-2) Labelled arrows or pointers define 
the relationships between the concepts, (f Inure 2-1) Other 
pointers indicate membership in general classes or specify 
particular properties, (figure 2-4) And pointers to circles 
representing the s tde s extend the depth of the description 
and allow wore detail, (figure 2-5) 

Now notice that notions like SUPPORTED -«V t ABOVE, LEFT- 
OF, BENEATH, and ft-KlWJ-OF nay be used not only as relations, 





FIGURE 2-1 








JJEF 



FIGURE 2-2 



LEFT -OF 




REGHI-yF 



FIGUEE 2-3 



LEFt-OF" 




FIGURE 2-4 




HGURfc 2-5 



2 3 

but also as concepts- Consider Sl!PPQP,TED-DY, The statement, 
"The WEDSE is SUPPORTER' -BY the BLOCH, 11 uses SUPPORT Ell -BY as a 
relation. Cut the statement, " SUPPORTED-BY is the Opposite 
Of NOT-S'J PPORTEu*EY* " uses SL' PPORTEQ -0Y as a concept 
undergoing explication. Consequently SUPPORTED -BY is a node 
in the network, as well as a pointer label, and SI/PPO&TEIJ-E Y 
itself is defined In terms of relations to other nodes. 
Figure 2-C shows some of the surround fnq relations and 
concept s. 

SUPPORTED -BY may therefore apoear in diagrams as a 
circle label or as a pointer label denendlnq r>n its function* 
A circle pierced by an arrow indicates simultaneous use as a 
relation and as a concept, (figure 2-7] 

Thus, descriptions of relationships can be stored in a 
homogeneous network along with t lie description? of scenes 
that use those relationships. This remits biq steps toward 
program generality, A. program to find negatives need only 
know about the relation NEGATIVE-SATELLITE and ha^e access to 
the general memory net. There fs no need for the program 
Itself to contain a distended table. This way programs can 
operate in many environments, both anticipated and not 
anticipated, Algorithms desianed to manipulate networks at 
the level of scene description can wort as easily wtth 
descriptions of objects* sides, or even of objects' 
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functions, q iv en the appropriate network. 
2..Z Prel im tnary Processing 

Consider now the Generation of a scene description* The 
starting paint 1s a line drawing without perspective 
distortion, and the result is to be a network relating and 
describing the various objects with pointers such, as IN- 
FROKT-OF, flBOV E , SU PP DATED -Blf, A -KIND -OF, ABUTS, and If AS- 
PROPER TY- OF . 

"irst, drawings of t hrcc -d fnen sional scenes are 
communicated to the machine uslnn: a prcoram by E + K» 1\ Horn 
together with a special pen whose position on a companion 
tablet can be read by t lie machine directly. Then a prooram 
written by H, N, Mahabala [1] classifies and labels the 
verteaes accord ino. to the number of converelnn lines and the 
anqles between them* ElourE 2 -8 displays the available 
categories^ r. c t i <: ^ that Mahabala's pro gran find's pairs of Ts 
where the crossbars He t-etwoen col linear uprights. These 
are called matched Ts, Such nairs occur frequently when one 
object partially occludes another as In fioure S-9. 

The program then proceeds to create names for all of the 
regions in the scene. THnorously "recr1on lh as used here sfnnly 
refers to any maximal area in which one can cove from any 
point to any other point wi thcut crossing a line, Includino 
the background f inure £-9 has el&ht reoions, Various 
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properties are calculated and stored for these regions. 
Amc?ng these are a list of the vertexes surrounding each 
region and a list of the neinh boring rPn1f:ns H 

These results are then supplied to the eleoarct program 
named SEE developed by A f Guzman [2]. This program 
conjectures about which regions belong tc the sane objects. 
For figure 2-§ t the end result of the program is the 
commentary: 

Body 1 consists of A Et C 
Body 2 consists of II f F G I 1 
Surpri sin-jl y the prograr contains r.o explicit models for the 
objects it expects to see* It simply examines the vertexes 
and uses the vertex classifications to determine which of the 
neighboring regions are likely to be part of the sare object . 
Arrows, for example. strongly suogest that the tra narrow- 
angle regions belong to the sare body, (f log re 2-10) This 
sort of evidence, together ulth a moderately sophisticated 
executive,, can sort out the r-enions in scenes as complicated 
as that in figure £-11, borrowed from Guzman's thesis. 
Twelve objects are reported and the regions of each are 
r em em be r ed . 

This, then* is the sort of information rejrfy for further 
processing by my description-building programs. 
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2 + 3 The Algorithms 

The follovnm suctions describe the idea s behind 
programs that look for the relations ACME, SUPPORT, PJ- 
FRONT-DF* LEFT, R Ifi h3T » and HARRVS* Generally these prOnr&nS 
produce descriptions that are in remarkable harmony with 
those of human observers. Sometimes, however, they make 
conjectures that most humans disagree with. On these 
occasions one should remember that there is no Intention to 
prex isely mimic psyc hoi oogf cat phenomena, The noal is simply 
to produce reasonable descriptions that are easy to fork 
with* Right now it is Important to design and experiment 
with a capable set of programs and postpone the ouestion o* 
how the programs might be refiner! to be more completely 
1 ifel ike. 

2,3,1 Above and Su pport 

T joints are str-ono clues that one object partly 
obscures another* but then one "flay as* if the- oh sour inn 
occurs because one obiect is above* the other or because one 
is in front of the other* Even in the simple two brick case 
there seems to be an enormous number of configurations* 
Figure l-\Z shows just a few possibilities* 

But fn spite of this variety, there is a simple 
procedure that often seems to correctly decide tne ABOVE 
■.■?-5ij- n- :u .i:r"-.T questior. Co^^r \*-# I'-h'^ t-:it *:— i 
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the bottom border of t he obscurinn objects in figure 2-1 2* 
Find ln-n these lines 1s the first job of the program. Next 
the program finds other objects whose renlons stare these 
1 1nes + In creneral these other objects are below the 
original, obseurine object. 

This algorithm works on all the simple two-blocfc 
situations, depicted Tn figure 2-12. It even worts correctly 
on the nuch rnorft complicated, many-object scene In 
figure 2-13, shown with the bottom lines highlighted. 

The difficult part is to find the so-called bottom 
lines, which correspond roughly to one's intuitive notion of 
bottom border. The process proceeds by first not inn those 
lines that lie bitten two regions of the object in nuestion, 
I call these interior lines. Next the prorrran nxa^fnes the 
lower of each interior line's vertexes. This Is ignored 
unless it is an arrow, X or a £. Then information about 
bottom lines is nleanec" from each of the arrows, Ks, and Ks 
In the fol low ton- way: 

1. Tf the vertex is an arrow* ther* the two lines 
forrtinq the largest angle, the barbs* are bet ton 
line candidates. See fiouro 2-14. 

2. If the vertex is an X, then the two non-coil inear 
lines are bottom line candidates. See figure 2»15. 

3. If the vertex Is a K» then the t^o adjacent lines, 
those forming the smallest clockwise and the 
snallest counter-clockwise angles with the Interior 
line are bottom line candidates. See figure 2-1 6, 
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This is really a rule and two corollar ies f rather than three 

separate rules* X s and Ks result primarily when arrows 

appear incognito^ catnnuf Iar?ed by an atfonment of objects as 

illustrated by fiqure £-15 and 2-16* Consequently, the 

corresponding rules anount to location toe arrow-forming 

parts of the vertex and then actino on that basic arrow* 

■One further sten is necessary before a line can become 

an approved batton*! fne* fs. shown by Moure £-17, Scene of 

tbe lines nualifyinq so far must be el In ina ted + They fail 

because they are too vertical, or p;ire orfclsely* because 

they ar£ too vertical with respect to the arrow's shaft. Tbe 

effective way to weed out bad lines is as follows; 

Rule: El imlnate any bottom line cand idate whi ch Is more 
vertical than the shaft of the arrow suggesting that 
caod frtat e, 

Of course the program extends rud imentary bottom lines 
through certain vortexes. Figure £-10 s^nws tbe obvious 
situations in which the bottom line property is extended 
throucih the crossbar of a T or the shafts of a pair of 
matched Ts, 

This whole algorithm is based on an assumption that the 
machine observes the scene from above. If the c r-nf Irrupt 1 or> 
dangles from the ceil inn, sinole changes adapt the rrnrriVi ta 
discriminate between UHPFrt and TN-FP.fiJjT-OF , rather than AB^VE 
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and IN -FRONT -OF. One examines instead the hioher vertexe.fi of 

the interior lines, substitutes the term top lines for bottom 

1 ines in the vertex inspection rules fc fl rtl the resulting lines 

usually separate objects from those above then. 

By searching for ooth the ABOVE and the BELOW 
relations, the machine may often be able to run?** its 
own heioht. Consider figure 2-19. Fir™ re 7.-2G shows 
the sar*,e scene with top and bottom lines h in hi in. hted and 
with the consequent above and belok relations* At least 
in this case t the machine can correctly deduce that Us 
eye is level with nb.iect y, because both a chain of above 
relations and a chain c* below relations originate at ? -» . 



2.3 J J D 1 scu 5 51 on 

This algorithm works effectively because of 
circumstances all likely but nnt certain to he true in any 
particular scene. The nethod works best when a seen* 
consists o* bricks and uiedfjes with one side parallel to the 
table* In many other cases, the method works any nay, 
sometimes by colnc fdence and sometimes by principles not yet 
fu 1 1 y e* pi ored . 

Unfortunately* in explanation I am frequently 
forced to appeal to intuitive notions about what is 
likely and what is not, I know of no way to establish a 
reasonable nroba hi 1 Ity metric on the situations I 
discuss. All that can be said nau is that any such 
metric should reflect human disposition toward 
c onf i nura t i on r ex n- -:i I. i- n c'-nnicnt ari s ■-■i "•;■*_ -v , 

The first 1 i te 1 y circumstance or urincinle fs that 

objects tend to support other objects b = y contact throunh 

relatively horizontal steles, nbiects nnt so supported tend 
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to slip, although not always as demonstrated by floure 2-21* 

Imagine now that the top object in figure Z-21 were 

completely transparent except for a layer of paint on the 

crucial* relatively horizontal bottom side, Since the icEne 

is viewed from above, this bared bottom aide will nbscure 

part or sometimes all of the supporting ot.ject. 5ec 

figure 2-?3. Consequently, some f aces of the swpportfng 

object nenerally border on lines resulting from the edges of 

the bottom s1de + 

Of course many ef the bottom side's ednes vanish when, 

the object is restored to Opacity* Nevertheless,, the ones 

that remain still tend to form part of the seam between the 

supported and the support fnn objects. 

Now most of the vertexes of objects are formed by throe 

edges meeting together as in the tip of a pyramid. This 

forms the s.e-calle.1 trihedral angle. Consequently, when two 

observable ertfies of the bet torn si rip form a concave an Me, r>ne 

can expect a third edoe of the object to leave the same 

vertex and form the shaft of a downward directed arrow, 

Slight alteration could permit the program to deal 
with nany objects v>.-fth non - 1 r i hod ra 1 vertices. The 
object In figure 2-24, for example, has twe Interior 
lines neraing at vertex V, By treatfnn this as a 
rieneraHzed arrow, wi t h mtjltfpte shafts, the same 
algorithm can be used to define hotton lines» 

So far the Ionic is as follows: If fine object ohscLpres 
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another because It is or, top of the other* then the sean 
between the two Is likely to form the barbs of one or more 
down uard directed arrows belonrrlnn to the top object. 
Reversing this, one would hoi>e for the statement! The barbs 
of downward dMrected arrows define seams across which one 
Hkely finds the support Inn object or objects. Unfortunately 
one often finds non*su poort Inp objects as well. 
2.3-1.2 Refinement 

Figure 2-25 suqoests a serious Mml of over enthusiasm. 
The pillar, brick C» elevates the bottom linns of brick fl and 
they appear between the viewer's eye and a side of the 
•Ms jive Lackorcund bricfc, brick C. In the two brick case, a 
brick's bottoms lines border on the side of another brick 
only when the one 1s in fact supported by the other* When 
more objects are involved, wronq answers may result because 
the bottom lines can wander into regions of objects that do 
not offer su pnort ♦ 

Tc handle this problem. I use a two part procedure* The 
first part is simply the above algorithm as described so far. 
which now may be thought of as oenerattno a set of possible 
supports. The second part, described below,, criticizes these 
possible supports and filters out some of the bad ones. 

Now one reason brick A 1n f Inure 2-25 clearly does not 
lie on brick C is that it is absurd to think that an object 
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can rest on a vertical side of some other object. Part Z of 
the support algorithm makes sure that just such conjectured 
supports are eliminated from the t 1 st of candidates. To do 
this* It eliminates a candidate if the wily bottom-! ine- 
horderlng side seems vertical. Jt assumes a side is vertical 
1f an edge belonging to 1t 1s vertical + 

There are two exceptions to this rci 1 e that Occur when an 
object obscures the entire top of its 5u r-port incr object as 
block P, obscures block G and as block B obscures block C ir 
figure 2-2G. First, If two bricks are aliened as are brfck 
A and brick a„ forming the familiar X vertex , ncv rejection 
takes place. Second, if the top brick overlaps the support 
as brick EJ overlaps brick C t at least one unmatched T appears 
and again no rejection takes olace. 

If by this time zero nr one support candidate remains, 
then part 2 terminates, and the support* 1f nny t is 
announced* There are some cemmon s it Jat ion s t however, that 
reouire part 2 to undertake additional C onpu tat ion * Comare 
figure 2-27 with figure £-20. The vertical -Side filter 
cannot eliminate the possibility of supoort from brfck C in 
either flours because one of C's bottom-line border f n rr sides 
is clearly net vertical. Vet human observers genera 11 y clflfm 
the top brick in figure 2-27 cannot bo other than singly 
suppcrted, whereas the> admit there may ks! 1 be supoort from 
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the Urrre rear Mock, C, in figure 2*£8. 

Since the only difference is In the he in tit of brick B, 
this judgement must be the result of a height eonparlslon. 
Stated simply, the program makes heinht judgements by 
assuming en sM«t 1* supported by the tallest of the support 
candidates survivtno so far. It Is simply above the others. 
The height cf a stack cannot always be computed rinnrrusly, 
however. In simple cases it is sufficient to locate * 
vertical line belonging to the object and measure it. Brick 
C in figure £-27 is such a Mack, fiut the verticals of bl«fc 
Lidlsarnear into T Joints and only mtn 1num he Hints can be 
calculated fron such lines without complicated and unexplored 
object extranolation techniques* Con sequent 1 y, as the 
algorithm reviews the heights and minimum heights presented 
to it, it first selects the maximum of these. Then any 
candidate vhose height is knani exactly is rejected 1* that 
height is less than the maximum just calculated. All whose 
exact heights are unknown are allowed to pass. 

Finure 2-29 shows why the support algorithm frequently 



ipnort for block C is to be 



f-Fsnrts to recur 5 ion. If the sui 
calculated, the heloht of blocks Q and E must be compared. 
Rut this In turn renuires knowledge of their support so that 
total heights can be added up fron the chain of supports and 
used in this last filter in a operation of nart ?. 
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This completes discussion of the support algorithm as ft 
now stands* It is not hard to delude it d el iberat el y, but it 
nevertheless operates with a reliability sufficient far use 
by programs that build upon its results* 

Inprov ement s can be tnade in many ways* The followlnq 
are a few Ideas hioh on my priority list: 1, A planned 
modification irvnlves usinn Ls In the search for bottom 
lines. So far the system gacs on the scene fn *inure 2-30, 
f i nd 1 n n no bottom lines* But if one line of an I Is nearlv 
vertical and the other 1s nearly horizontal* then the nearly 
horizontal line should be an excellent bottom line. Of 
course Ls ere frequently burled just as arrows arc, Burled 
Ls are found in certain Ts, Xs and forks *s shrsv-n in 
figure 2-31. ?* The discovery of bntton lines can be fouled 
by introduced small objects that obscure crucial verte*es. 
f inure 2-32 shows how. This could be corrected by a 
procedure that extends lines, perhaps after the object is 
identified. 3, The filtering operation could be strennthenod 
io its ibllfty to detect vertical sides* So far it knows a 
side is vertical only if the side has a vertical &dne, 
Figure 2-33 shows how it can run aground as a result, fiont 
of the sides -of brick C appear vertical and the algorithm 
consequently reports brick A is rr, ton cf brick C as well as 
on brick B. 
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2.3.2 In-front-of 

Once a program can discover the SUPPORTED -GY relation, 
then U can frequently deduce IN-FR0NT-0F relation* by 
default* That ls + If one of two blocks appears to obscure 
another but is not above ft, then the relation IN-FRONT-OF is 
a strono possibility. In pursuing this I strain use a two 
part programs the first part proposes possible objects that 
a given object may be in front of; the second part rejects 
the bad ones. While simple and direct, this program also 
succeeds admirably en comple* scenes.. 

Part 1 tries to ffnd all objects that the object in 
question obscures. First it gathers up nest of the obscured 
objects through search for particular types of T joints on 
the periphery of the object* Supoose one defines a line to 
be physically associated with a particular object when that 
line in the two dimensional drawlnu results from an edge or 
intersection of planes on tbe object,, rather than from some 
obscuring object. Figure 2-34 Illustrates* Then the types 
of Ts sought arc just tnose for which cne can be reasonably 
sure that the crossbar belongs to the object conjectured to 
be the obscuror t 

figure 2-3 5 shows two kinds of qualifying T joints* The 
first kind occurs when the wide angle region associated with 
the T belonqs to or is physically associated wftb the object 
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from whicn ri -FROM -OF relations are sought. Both of the 
other recions, the ones border ing the shaft of the T> be loner 
to a second object. This nearly always Indicates the second 
ohject Is obscured. 

The second Vind of T joint Illustrated in figure £-35 
occurs when one shaft-bord er int? region belongs to nn object 
while the other belongs to the bac heron nd ♦ This again 
assures the machine that the crossbar belonqs to the 
potent tally obscuring object. 

Figure 2-36 indicates by counter example why nothing can 
be deduced if the shafts border ing regions belong to distinct 
object;. The trouble 1s that the crossbar of the T 1s not a 
real edge of Mock; C* but rather the crossbar is composed of 
edges belonging to A and B, 

Still another way to locate appropriate Ts is more 
global- The Idea is to use whatever means are available to 
find genuine edges belonging to a body and then to see if any 
Ts 1 ie along such lines. Recall that the selection of bottom 
lines Involves Inspection of arrows, X s and Ks at the bottom 
ends of interior lines. Furthermore the rationale behind t l*e 
support algorithm depends on the likelihood that such bottom 
lines are physical 1 y a s soe la ted with the same object as the 
interior line of the arrow, X , or K. Consequently the machine 
can generate a whole family o c peripheral lines likely to be 
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physical edoes by simply examining the arrows, Ks, and Ks at 

both ends of the interior ltnes T rather than Just those at 
the bott&nt ends* Then if any of these physically associated 
lines end at an L, the other line format] the L is added tc 
the list. It Is very unusual for one leg of an L to belong 
to art object without the other lee belonging also* 

flow if any physically associated line 1s the crossbar c^ 
a T ± then the parent object obscures sone ether object or 
objects. Moure 2-37 demon stra te s what kind of edoes and Ts 
are found by this method. 

Notice that the Ts referenced by figure 2-37 are also 
noticed by the previously discussed 1 oca.1 Inspection since 
they exhibit the rehired c on f inurat f on of objects about the 
Ts lines. Figure i^ -38 demonstrates that both methods 
contribute, however, since only the local method worlks on Ts 
marked l_ while only the modified ho ttomj-1 ine finrfer helps on 
those marked G * 

All of this yields obscured objects which are candidates 
for relating to the object studied by the relation IH -FRONT* 
flF* riejtt, part 2 requests help from the support program and 
then famed lately rej ects all the candidates for which the 
A^OV E relation is known to hold, This however is often not 
completely sufficient. In figure 2-3J, the machine trows 
bricfc A obscures brick C by virtue of vertex V, But ft is not 
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directly above C, Clearly* the above check ng st be expanded 
to the presence of chains of A&0VE relations In order to 
bring t he algorithm into line with human taste* In the 
course of this check, the program for AfcflVE may be called 
many times if the ABOVE relations have not yet been 
established, Any candidate that survives this check is 
thought to be behind the object studied* 

The most annoying weakness of this algorithm "fs that the 
seam between the obscured and the obscuring object may not 
exhibit the required type of 7 joint?, Figure 2-4D shows how 
this can happen. I suspect that further progress can be made 
in these situations of alignment through close consideration 
of Xs and perhaps Ks. 
2.3,3 An Example 

Figure Z-fll provides a somewhat more ccmoleH scene for 
the IN-FRONT-OF and 5U PPOP.TEP-B^ finding orograms to try. 
The results are as follows: 
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The only b-a d choice is t he neglect of C as a su poort of F* 
The reason is that the support criticizing nroc-ran has a 
built in assumption that a supported object's bottom Is 
level. Therefore it believes E is the only support for F 
because 1t is higher than ft, the other possibility. 
2*3*4 Left and Right 

Tko programs exist for deciding if one object is left 
of, right of, or neither with respect to another. The first 
computes in a straightforward, simple way. It simply 
conpares the x coordinates of the vortexes of both objects. 
If there is no overlap, that is if 
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xccr{any vertex, of one object) 

< 

xcor(any vertex of other object) 

then the first object is to the left c* the other. If there 

1s overlap, t*"ier no statement ca ti be made. 

This program, based on th# no-cverlap criterion, could 
be greatly Improved through the use of an object extending 
program. The machine Is nai^e to think trat object C in 
figure 2-42 la left of object A. Hunan s tend to fill in the 
obscured portions of object C to form a complete- block. 

Ryt even vith an &l;1lity to Imagine the hidden parts of 
objects, such a program refuses to really arrefl wf t h human 
judgements. Consider the spectrum of situations in 
figure 2-43. For the first pair of objects, thE relations 
LEFT-OF and RIGHT-OF are clearly appropriate* For the last, 
they ar^ clearly not appropriate. To me 4 the crossover point 
seems to be between the situations expressed by pairs 4 and 

Now notice that the center of area of one object 1s to 
the left qf the 1 ef truest point of the other object in those 
cases where LEFT-OF seems to hold. It is not so positioned 
If LEFT-OF does not hold. Such a criterion seems 1n 
reasonable agreement with intuitive, pronouncements for many 




FIGURE 2-12 



s> 



r^J 



i=b 



c^ 



(^ 



d 


^j 




■■ 


^-tD 








^ 


t= 


=3 




r 


— U 


i> 


tz 


-D 




e 


-Jl 





6 



FIGURE 2-43 



63 



of the cases I have studied. It also yields reasonable 
answers In figure 2-44 where in one case A is to the left of 
and 1n the other esse it is not, Notice that the relation 
Is not syitmtetr 1c, however, as the. center of area of the much 
longer brick, brick B, Indicates B is to the right of f< in 
both cases. 

Figure 2-4.5 requires extra attention, No matter v ha t 
the center of mass relatinns, huiran s. are reluctant to use 
either LEFT-OF or B IGHT-OF if one object extends beyond the 
other in both directions. One must additionally specify a 
rule against this, leaving the following for LEFT-DF: 

Say A is left Of B <=> 

1. The center of area of A is left of the leftmost 
point of B, 

2. The rightmost point of A is 1 #f t of the rightmost 
point of B + 

The rule for 'UGHT-CF 1s of course parallel in form. 

Many people feel their perception nf the relation 
LEFT-OF differs considerably from either Of the two 
possibilities exhibited here b I believe the tenter^of- 
area method is reasonable for the machine now, but it 
would be interesting to more fully explore the Question 
of what humans think to see if other formulas are 
better. Intuitive notions of LEFT-OF vary wildly and 
the program can only ce said to generally reelect my 
personal preferences. Indeed, deciding if one object is 
to the left of another stimulates far more argument than 
do questions involving relations like IK-FRONT-OF and 
SUPPORT ED -BY. People have difficulty verbalizing hoi* 
they perceive LEFT-OF and tend to waver in their 
methods, but Implications are that criteria change 
depending on whether the objects involved are also 
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related by IN-FRQ.NT-DF, ON -TOP- OF, RIGPEP-THAM f and so 
Oft* 

Professor M&rv f n ft In sty has pointed out to mo that 
the orientations of objectsare also a strona influence. 
In figure 2-46, for example* the cube seems left of the 
arch" 5 entrance even though all its vert exes are Clearly 
right of all the arch's vert exes* In vfew of this 
observation* ny procedure could probably do better by 
asMne basically the same questions as be fore » but about 
lines through the left-most* r f ci ht -nost, and center-of- 
area points in the direction of orientation instead of 
what amounts to vertical projection of the points to the 
x-axis-, Put then there is the problem of finding an 
object's intrinsic orientation, At the moment I know of 
no general heuristics for this, 

Z.3. 5 Harrys 

The abuts and a1 Igned-wlt h relations arise frequently, 

perhaps because of some human predilection to order, As 

intuitively used* however, neither of these *ord s corresponds 

to the notion I want the machine to deal wftlu To avoid 

confusion* I therefore prefer to usft the torr marry which I 

define as fol lc»ws: 

Definition: An object marrys another "If those objects 
have faces that touch each other and have at least one 
common edge. 

Thus the objects "fn figure 2-47 are said to marry one 

another . Those 1n fierure 2-43 do not because they have no 

common edge, Similarly those in figure £-49 do not because 

they have no touching faces. 

The HARRYS relation is sensed by methods resembling 

those previously described. First the vert exe s al ong the 

border are collected. Then the Xs» Ts and Ks are further 
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examined: 

The Xs are s im-p 1 e to handle* If exactly two of the 
lines are collinear and 1f the other two separate two 
objects, then those objects are very likely to deserve the 
MARRYS relationship because such a vertex is strongly 
associated wfth aligned stacks or rows. Mtrure £-50 
Illustrates these situations* Figure 2-51 shows why the two 
objects rtiu st be separated by the two non-col l linear lines* 
There three sides belong to the saroe object and the riARPYS 
relation does not hold. 

If there are four objects at the vertex and there are 
twq pairs of collinear lines, then the likely situation is a 
field of objects with those sharing 1 1 r tv s marrying each 
other, See figure ?-5£. 

Ks are strongly correlated with the sort of alignment 
Illustrated by figure 2-53. The rule is slnnler If there 
are two objects with sides at a K vertex, then they probably 
marry* 

When three objects meet at a T one of the foil owing 
hold s: 

1* The objects with the stem in between marry each 
other and both obscure the third object* 

2, The third object is the obscuring object. In 
th-f 5 case the two Other Objects may Or nay not 

marry. 





















j^ 
















FIGURE 2 


-52 








FIGURE 2-53 




FIGURE 2-5*1 



71 

3« All three objects marry. 
4. Something el se . 

Of these, ny programs check for case 1 only* Thfl 
central program looks for chains of IW-FRQtJT-QF and SUPPORTS 
relations between the shaft -bord eMng objects and the large- 
angle object. If such chains are found for both, they likely 
both obscure the large anqle side and marry each other* 
Figure £-54 shows examples. 

Now consider the situation where only two objects meet 
at a T with the wide angle side and one of the other sides 
belonging to one object. As fioure 2-55 sunrjests. this sort 
of T frequently becomes a K when seen fron some other angle* 
Like the K t the machine considers it strong enough evidence 
for a r'ARRYS relation. 

Figure 2^56 illustrates both of the T joint situations 
that confirm marrymoTit* 

Jlote that the machine fs conservative 1n using t^is 
MARRYS relation; the relation is not placed In ambiguous 
situations Such as those of figure Z-57 . 
2,3 + 6 Shape 

Before an object can be identified, the lines that are 
really edoes of that object must be sorted from those that 
are edoes of obscuring objects* It would not do to think 
object 6 in figure 2-58 has sides shaped like those in 
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figure 2-59. 

The Idea of an edge belonging to a body ha? been 
discussed. The shape oroqram I use first gathers together 
those lines found to be genuine physically associated edges 
by the program that looks for- IN -FRONT -OF relations* To 
these ft adds any tines that lie between two regions of a 
body, the interior lines* Thsn if these 1 -Tries include both 
uprights in a pair of matched Ts, it adds a line joining the 
two Ts. And finally, any line shared with the background, or 
other body known to be he low or behind is certainly Included, 
These rules are sufficient to Identify many of the lines that 
belong to any given object f while rejecting many that do nnt 
belong. 

Figure 2-GO shows how this program sees object 6 of 
figure 2-53. Lines L t M, N, and Pare interior lines* Line 
Pisa segment between matched Ts with the required kind of 
uprights, Q cpjallfies by way of the IN-F^ONT-OF algorithm* 
while R f S, T, J, and 7 qualify both by kay Of the IN -FRONT- 
OF algorithm and the rule adding lines lying between the 
object and the background* Figure 2-6 1 shows how the rest of 
the scene in figure 2-5B is dissected by this program. 
Notice that thp shapes are reasonably well defined. 
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2.3*7 Size 

Piaget has shown that at a certain age children 
generally associate physical slie with greatest dimension 
[3]. They wiH, f or exanrle, adamant 1 y ma inta in that a ta 11 
thin beaker has more water in it than a short fat one even 
though they have seen them filled from other beakers nf equal 

size. 

Adults do not develop as far beyond this as might be 
expected. I do not think we really use the notion of volume 
naturally. Apparent area seems much more closely related to 
adult size judgement. Notice that beaker A in ftgure 2-62 
appears to have about the same amount of water in ft as does 
beaker B, even thouah ft mu st contain twice as intich* Unless 
a su bj ec t consc iously exercises a formula for volume, he is 
lUely to report that object B 1n f fog re 2 -S3 1s 
approximately ten tines larger than object \, even if told 
both objects are cubes. The true factor of twenty-seven 
times seems large when the trouble is taken to calculate it. 

Consequently* the size generating program does not 
trouble with volume. Instead it calculates the area of each 
Shape produced by the shape detecting algorithm. Next it 
adds; together the areas of all shades belonging to an object 
to get its total area* Then using these areas it can compare 
two objects in size or consult trie following table for a 
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reasonably believable discrete part it ion 1nq of the area 
sc a 1 e : 
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3 Discovering Croups of Objects 
When a scene lias more than 4 few objects, it 1s usually 
useful to deepen the heirarchy of the description by dlvidinn 
the objects. "Into small or groups which can he descri be ri and 
thought of as individual concepts, Figure 3-1* for example! 
seems to divide naturally into three groups of objects, one 
being three objects tied together by SU PPDITTED-EY po in tors, 
another being three similar objects on top of a fourth, and 
the third be Inn a set of objects 1n the arch, conf 1 jurat 1 on* 
There are other kinds of grouping humans use, but in this 
work T primarily explore only the three illustrated by this 
figure 3-1. Grouping by Identification with a known rndel 1s 
discussed later in the chapter on identification. This 
chapter deals with grouping on the basis of pointer chains 
and on the basis of property similarities* 
3 , t Sequences 

A simple kind of group consists of chains of SUPPORTED - 
BY or IN-FRONT-GF pointers as 1n the tower of figure 3-1. 
The first act of the grnupina program is to find sets of 
objects that are so chained together. All such sets with 
three or more elements ouallfy as groups* 

In the event the sequence of pointers closes on itself, 
a ring is formed. In figure 3-? there Is such a group 
because each of the three objects rests partly on ore of the 
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ether t*0* The result is a circular chain Of SUPPQRTED-BY 
pointers as shown in figure 3-3 + 

tiling chains to define groups can become fairly complex 
as Illustrated by the scene in f inure 3-4* A chain of 
SUPPQRTED-BY pointers splits into two branches in scene one 
at the point where object C is supported by two objects, D 
and E. In scene two* two chains of SUPPORTED-&t pointers join 
at M which supports both I and L. The current version of the 
grouping program terminates chains at junction points without 
further fuss. This seems reasonable for it seems natural to 
think of the scenes in fioure 3-4 as a set of groups 
consisting of A-B-C, G-k-l, and J-K-L. 

Another kind of nroblem arises when objects tied 

together Ly a simple chain of relations should not 
constitute a group because of other factors. Figure 3-5 
shows one kinrt of situation that can occur* for which I 
have only ideas but no programs* In this scene the 
machine perceives a sinale abject conglomerate, grouped 
together by virtue of an unbroken chain of 5U PPORTED ~BY 
pointers* But most human s see a short tower on top of a 
board on top of another tow?**. This must be partly 
because of the size differences and partly because of 
the fact that the top group is. not directly over the 
other objects. In any case, it would seem that radical 
change 1n object properties should be oossible grounds 
for breaking a chain* With this, one is Into territory 
where Irrevocable committments should be avofded* 
Perhaps the best thing would be to have the Grouping 
program offer alternative groupings of tricky scenes and 
postpone rfec is ion until higher level identification 
programs indicate which arrangement leads to the best 
match of the scene against known models. 
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3.2 Common Properties 

When several cbjects have the same or very nearly the 
same description, they are fmmedlatsly solid candidates x or a 
group. The lees on the table in figure 3-6 arc typical* All 
are bricks, all are standing, and all are supports for the 
top board + 

TMs kind Of manipulation is slightly dangerous in that 
my criteria for forming a group and admitting members to it 
are a bit flimsy* So far the ruTes are based on the 
foil awing demands: 

1. All candidates for group membership hluSt be- relatEd 
to one or more particular objects in the saire nay* 
Fcr the table case* all four objects are related to 
the heard by SU PPOftTEU-B V, This restriction appears 
necessary because oniforn relationship to a single 
object seems to have strong binding power. The 
standing bricks in figure 3-7 naturally constitute 
two groups, not one. 

2. There must be three or more members 1n the group* and 
the members of the group must share many of their 
properties. 

Figure 3-3 outlines the procedure for forming such 

groups* The basic idea is to make a generous guess as to 

what objects to include in a group and then to eliminate 

objects which seem atypical until a fairly homogeneous set 

rema ins t 

To do this* a program first finds a c-ndfcfate group by 
locating a set of objects that relate to one particular 
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object in the same nay + Ne*t comes formation of a corrnion- 

relationships-1 i st through a listing of all rela t lonshi p s 

exhibited by more thin half of the candidates 1m t'-.e set. 

Figure 3-9 helps explain this process* Objects ft 

through F are immediately perceived to be a possible group 

because they all have a relationship, 5U PPORTED-BY, with a 

Single object, G. The relationships exhibited by the 

ca nd Ida tea ar er 

A, B» and Cj 

1 SUPPORT ED -BY pointer to G 

1 HARRYS pointer to G 

3 A-KIND*OF pointer to BRICK 

4 HA5-PR0PERTY-0F pointer to HEDIUK-STZE 



O: 



■ 
2 
3 
A 

E and F: 



SUPPORTED*!} Y pointer to G 
MARRYS pointer to G 
A-KINL-OF pointer to RRICK 
HAS-P* OP ET:T V-OF pointer to SMALL 



1 3U PPtfRTEC-BY pointer to 6 

Z MARRVS pointer to G 

3 A-KIND-OF pointer to HEDGE 

4 HAS.PROPERTY-OF pointer to S^ALL 

Three relations appear in the conmon-relat ionships-1 1st 
because they an found 1n more than half of the candidates' 
relatlonsh ips 1 i st s: 
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common -T 1 el atlonshlps-list: 

1 SUPPORTED -BY pointer to G 

2 MARRYS pointer to G 

3 A-KIIJD-OF pointer to BRICK 

After this, common-relat ionshlp s-1 f st is formed, all 
candidates are next comoared with it to see how typical each 
Is. The measure 1s simply the fraction of the total number 
of properties of the candidate and the common-relat lonshtps- 
1 n st that are shared* Said In a more formal way, the measure 
1s 



number of properties 1n Intersection 
number of properties in union 



where the union and intersection are of 
the candidate's relationships list and 
the common-relat 1 on ship s-1 1st* 



Figure 3-10 represents abstractly a situation fn which the 
candidate and the cammon-relat ionsh ip s-1 i st are quite 
different. The shared properties* represented b-y the shaded 
area, 1s but a very small fraction of the total area, both 
shaded and unshaded* Figure 3-11 gives the opposite extreme* 
There 1s considerable overlap and the value is near oine, the 
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max imum pOSSl hi e. 

Using this similarity formula to compare the various 
objects of the figure 3-3 exarple with the common- 
relation shlps-1 1st , one hasj 

A verius the c-nmmon-rela t1o nshlp s-1 i st --*> 3/4 = -75 
C versus the tonwi on-relat ■? on sh fps-1 i st - -> 3/4 ■ »75 
C versus the cnmnon-relat icnshlp s-1 i st --> 3/4 = t 7E 
D versus the coinnon-relat ionshlps-1 i st — > 3/4 = *75 
E versus the common -r el at ion ship s-1 1 st --> 2/5 ■ .20 
F versus the common-r e la tionship s -1 1 st --> 2/5 = .20 
A, B, t. and D do net have scores of 1 only because the 
common-relationshlps-fist does not yet have a property 
Indicating size. The reason 1s that there is no size common 
to more than half of the currently possible group members, A» 
Q t [, D, E, and F* 

The much lover scores of E and F reflect the additional 
fact that as wedges they are different from the standard 
type. They are immediately eliminated according to the 
following general ruler 

Eliminate all candidate objects whose similarity scores 
are less than 8C£ of the best score Any object attains, 

This insures that the group will have members all vrith a 

nearly epual rfght to belong. 

Next the process is repeated because those properties 
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common to the remaining candidates nay differ from those 
properties common to the original group enough that one or 
mor e c hanges should be made to the comnon-r el at f on s h 1 p -s -1 i st . 
This repetition continues until the elimination process fails 
to oust a candidate or fewer than three candidates remain* 

After the first elimination of objects leaves A* B> C* 
and 0, there Is a new common -re la t ion ships -1 1st: 
comnron< , re1at"£on'5h 105-1 i st : 

1 SUPPORTED-GY pointer to G 

2 tfAR^YS pointer to G 

3 A-KIND-CF pointer to BRICK 

4 HAS-PROPERTV-OF pointer to MEDIUM -SIZE 

Notice that there is now a size property since three of the 

four remaining objects have a pointer to medium size. The 

new comparison scores are: 

A versus the comncn^rel at ion s hip s-1 ist --> 4/4 = 1 
B versus the common-rela tion ships-1 i st --> 4/4 = 1 
C versus the common -relation Shi PS-1 ist --> 4/4 = 1 
U versus the coirnon *relat f on sh tp s*1 ist --> 3/5 ■ .6 
This time l> is rejected because its uncommon size causes 

4 low score* leaving a stable group in which the objects are 

all suite Hke one another* 
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3.3 Other Kinds of Grouping 

There obviouily cannot be a single universal grnouplng 
procedure because attention must be paid not only to the 
scene Involved, but also to the needs of the various programs 
that may request the grouping activity, I have discussed two 
grouping modEs that programs can now do in response to 
various demands* There remain many others to be explored, 

One of these involves looking for thing* that fit 
together* Children frequently do this at play without 
prompting, and adults do -ft extensively in solving jigsaw 
puzzles. 

Another kind of grouping, one oarticularly sensitive to 
the goals of the request, is grouping on the basis of some 
specified property* The idea Is to pick out all things 
satisfying some criteria* such as all the big standing 
bricks* The result could he a focusing of attention. 

Still another way to group involves overall properties 
that are not obvious from purely local observations* 
Techniques here are again lamely unexplored, but it seems 
that overall shape can sometimes impose unity on a complete 
hodge-podge. Figure 3-l£ Illustrates this point, All of 
the objects fit together to form a brick-shaped group* This 
is clearly not inherited from aoy consistency in ho*- the 
parts are shaped or how they interact with their immediate 
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neighbors. 
3*4 Describing a Group; The Typical Member 

The irachine needs some means of describing groyps. The 
method it uses seerns to worki but there Is room for 
impr dv enent. 

First, the parts of the crroup 3f*& qathered together 
under 3 node created spec If lean y to represent the group as a 
corceptual unit. Figure 3-13 -illustrates this step for a 
group of three objects, A B and C* 

Next cones a ccrcise statement of what membership in the 
group means;* This is eione through the use of a typicaT- 
member node* Properties and relations that most of the group 
members share contribute to this node's description. If some 
group were composed of three stand Inq bricks arranoed 1n a 
toiler, then the result would be the description shown in 
figure 3-14, The typical member is there described as a kind 
of brick, as standing,, and as on top of another member of the 
group. Notice also the FORM pointer to SEQUENCE which 
indicates the kind of group f ori*ied - 
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4 Similarities and Differences 
4.1 Network Matching 

Powerful scene description proqrams are essential to 
scene c ompar f son and 1d en t if icat ion* Matching is equally 
■important since the machine must know which carts of two 
descriptions correspond taefore it can compute similarities 
and differences. Figure 4-1 briefly Illustrates* A process 
explores the two descriptive networks and decides which nodes 
of the two best correspond in the sense that they have the 
same function In their respective networks. The nodes in a 
pair that sc correspond are said to he linked to each other. 
The job of the matching program is sirply to find the linked 
pairs* Node LC and node RC in figure 4-1 both have only A- 
KIMD-OF pointers to BRICK* Since no other nodes have similar 
descriptions, it is clear that LC and R C should be a linked 
pair* Similarly^ LS and SB should be a linked pair since 
both have A-KIND-OF pointers to HEDGE and both have 
SUPPORT ED -BIT pointers to parts of a pair of nodes already 
known to be 1 in ked . 

Of course the job of the matching program 1s not SO easy 
when the two scenes and the resulting two networks are not 
identical, In this case the process forms linked pairs 
involving nodes that may not have identical descr ipt ions t but 
seem most similar nevertheless. More details are described 
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in the appendix* 

4.2 The Skeleton 

Once the matching process has examined twc networks and 
has established the linked pairs of nodes*, then description 
of network similar ft fes proceed 5 „ The result is- simply a new 
chunk of network that describes those parts of the compared 
networks that correspond. This chunk 1s called the skeleton 
because it 1s a framework for the rest of the comrar i sen 
descr i p-t f on - As figure 4-2 suggests, each linked pair 
contributes a node to the skeleton. Certain pointers connect 
the new nodes together* These occur precisely where the 
compared networks both have the same pointer from one member 
Of some linked pair to a member of some other linked pair. 
Rotlce that the skeleton 1s basically a copy of the structure 
that the compared networks duplicate* 

4.3 Con par i sen r-.gt h^> 

Complete comparison descriptions consist of the skeleton 
together with a second group of nodes attached to the 
skeleton like graues an a crape cluster, Each of the nodes 
1n this second category is called a c-note, short for 
comparison note. The most coirmon tyoe of c-note Is the 
intersection c-note which describes the situation 1n which 
both members of a linked pair point to the same concept v1th 
the same pointer* Suppose* for examole* that a cafr of 
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corresponding objects from two scenes are both wedqes* Then 
both concepts exhibit an A-KlND-OF pointer to the concept 
WEDGE, (figure 4-3) In English one can say: 

1. Trier e is seething to be said about a certain 
1 in ked pa ir. 

2. "here is an intersection Involved^ 
3* The associated pointer is A- KIND -OF. 

■'> . The intersection occurs at the concept WE)G E« 
Figure 4-4 shows bow each of these sinnle facts translates to 
a network entry, First, a pointer named C-KOTE extends frum 
the skeleton concept corresponding to the linked pair to a 
new concept that anchors the intersection description. The 
A-KIHO-OF pointer identifies this concept as a Mnd of 
Intersection. Finally other pointers identify the pointer, 
A-KIND-PF, and the concept, HEDGE, associated with the 
intersec t ion . 

AM of the c-notes look like this intersection paradigm, 
,3,1 Digression: Evans' Program 

Embodying difference descriptions fn the same network 
format permits operation on those descriptions with the same 
network programs* Thus two dffference descriptions can be 
con pa red as handily as any other pair of de scr fDt fons. 

Those familiar with Tom Evans" vanguard program, ANALOGY 
[4] t can understand why this is a powerful feature, rather 
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than simply a contribution toward memory homogeneity* Evans" 
program worked on two dimensional geometric figures rather 
than drawings of three d imercs forva 1 configurations. 
Nevertheless his ideas generalize easily and fit nicely Into 
the vocabulary used here. 

Figure 4-5 suggests the standard sort of Intelligence 
test problem involved. The machine must select the scene X 
which best completes the statement: A i s to B a s C is to X* 
Jn human terns one must discover how B relates to A and find 
an X that relates to C 1n the same way. 

Using the terminology of nets and descriptions, one 
solutfon process can be formalized In the following way: 
First compare A with & and denote the resulting comparison- 
describing network by 

d[A;R]. 
Similarly compare C with the answer figures generating 
descriptions of the form d[C:X]. The result is a complete 
set of comparisons describing the transformations that ttrry 
one figure into another. Next one should compare the 
description of the transformation from A to B, d[A:E], with 
the others to see which Is most like 1t, The bast match is 
associated wild the best answer to the problem, if M is a 
metric on comparison networks that measures the difference 
between the compared networks, one can say 
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choose, X such that 

H(d[d[Ar8]sd[C:X]J) 

1s minimum 

The metric I u se 1s not fancy* It Is the one discussed 
later in chapter 7 that serves to Identify socie scene with 
some member of a group of models* It 1 works because that 
problem entirely parallels the problem of Identifying a given 
transformation description with some member of a. group. The 
identification program, together with a short executive 
routine, handles the problem of figure 4-5 easily, correctly 
reporting scene three as the best answer* Reasonably enough, 
the machine thinks scene one is the second best answer* 

The machine does as well on the slightly altered problem 
In figure 4-£* report Hg ^our as the best answer* 

Of course 1f the machine's answers are to be those of 
the problem's forniuTator* then the machine's describing, 
comparing*, and comparison measuring processes should all jive 
results that resemble his. Moreover, a really good analogy 
program should ha</e available alternatives to its basic 
describing, comparing* and comparison measuring processes. 
Then in the event no single answer Is much bet to** than the 
others, the program can try some of Us alternatives as one 
or more of its basic functions must not be operating 
according to what the problem maker intended* Evans' program 




A 



Jz^> 



B 



71 



• 



y ^1 




ONE 




TUO 



THREE 





FOUR 



fiue 



FIGURE 4-6 



11 1 



is Superior to mine -fn this respect because It can often 
compare two drawings 1n mo re than one way. It can visualize 
the change in figure 4-7, for example, as either a reflection 
or any one of several rotations. 

Given my formulation of the analogy problem, it is easy 
to see howcertafn interesting generalizations can be made. 
After all, once an X is selected, the network symbolized by 
d[d[A:&]:d[CiX]] describes the problem,, and as a description, 
it can be compared with the descriptions of other problems, 
Ely thus applying the comparison programs for the third time, 
one can deal with the question, Analogy problem alpha is most 
like which other analogy problem? A - ternat ivety, one can 
a a ply the analogy solving program to pror.1 en descriptions 
instead of scenes and answr the question, Analogy problem 
alpha is to analogy problem beta as analogy problem gamma 1 s- 
to which other analogy problem? This involves four levels of 
comparison* But of course there is no limit, and with time 
and memory machines could happily think about extended 
analogy problems involving an arbitrary number of comparison 
1 evel s, 
i + 3.2 Another Digression: Newell, Shaw, and Simon's Program 
A classic piece of work in artificial intelligence; is 
that of Newell » Shaw, and Simon on the scheme known as the 
General Problem Solver, always abrevlated GPS [ S-] , One form 
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of GPS provides another example of how comparisons may be 
usefully compared with Other compart sofls, 

From a ni abstract point of view, GPS Involves the notion 
that problems may be thought of in terns of some solution or 
goal,, t , together with a current state C* Additonally there 
are operators* 0(1)* that concert clashes of states into 
others Or, e may abrevlate t h& ir action by writing 

C(i):F-ItU1] --> F-0UT(1J, 
Meaning that operator 0(i) tends to convert states of the 
form F-IN(1) into states of the form F-OUT(i), 

GPS notices the difference between the current state C 
and the desired statn G and then tries to apply an Operator 
relevant to reducing that difference. This nroduces a new 
current state scwnewnat closer to the desired state* Applying 
this process iterativcly, GPS may eliminate the difference 
between C and B„ thereby solving the problem, 

In early versions of r*PS the programmers supplied a 
table giving the relevant operations for all the differences 
between C and G that might be observed. But later on Newell 
described an approach [6] that 1 think may be more 
transparently represented Using the sane notions of second 
order temp ar f st>n minimization that is useful in discussing 
analogy problems* The idea is that the operators, 0(i}» n?y 
be described! by the difference between their Input and output 
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forms, 

Then Newell feels it is hour 1st teal ly sound to apply the 
operator whose description is mest MU the difference 
between the current and desired states, 

d[Csfi]. 
One can say more formally, 

choose the operator C(i) such that 

H(d[d[F-IN{i):F-DUT(1)3rd[C:G]]} 
1s mill IflUW. 
Notice that the selection of an operator fs curiously 
like solving an analogy problem fcr which one chooses a pair* 
(X(i} t Y(1)) from a set of offered pafrs that best completes 
the statement: A is to B as X ( 1 ) is to Y[i). 
4*4 A Catalogue of C-note Types 
4.4,1 The Supplementary-pointer 

Consider the scenes in figure 4-8 and their descriptions 
in figure 4-9* Scene L has the pointer SUPPORT ED -BY between 
LA and LB, but scene R dees not have a oofnter between the 
objects linked to LA and LB, The note describing this 
situation is called a su ppl ementary-pcinter c-nete and has 
the form shown In figure 4-1 + 

Figure 4-11 suggests a related situation, here the 
linked concepts I and R differ only in tbat L has an 
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additional pointer Identifying 1t as startling. This differs 
from the su ppl em en tar y -pointer case in that STAND I KG is not 
linked to anything in the other scene, A pointer to the 
concept EXIT signals this situation, {figure 4-12) Exits 
Involve concepts generated by the scene description program 
as well as concepts like STANDING that reside In the net 
permanently* If one scene contains more objects than 
another, the concepts left o/er and net matched end up in 
exit packages. 
4.4 t 2 Pointer Modifications 

Suppose the networks in figure 4-13 are compared. 
Notice the MARRY5 pointer between LA and LB and the £)0ES*flOT- 
MARRY pointer he t ween RA and RB. These could be handled 
individually as unrelated supplementary-pointer c-noteS t but 
this would ignore the close relationship between HARRYS and 
DDES-NOT -MARRY, Consequently a different type of c*note is 
used that recognizes the relationship* It is the negative- 
satellite-pair c-note. With it, the comparison looks as 
sheun in figure «-U, To find suc^i negat ive- sat el 1 1te-pa ir 
C-notes, the comparison programs peruse the descriptions of 
unmatched pointers between linked pairs for evidence of 
relationship* For example, MAftRYS is described In part by a 
NEGATIVE-SATELLITE pointer to L)0ES-HOT-MflRY. Now of course 
there are other pointers that are also just one step removed 
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from 4 basic relation, Ml such pointer? that are 
mod Iff cat fens of the basic relation are called satellites 
because they cluster around the basfc relation to which they 
are attached by t he pointer MOO IF IC AT I0H -OF, Uncertainty, 
for example, 1s expressed by PROBABLY satellites or MAYBE 
Satellites. The MUST sa tell ites and thsMUST-NOT satellites 
are others of particular importance fn Tnodel construction* 
These inform the model matching programs that the presence or 
absence of some pointer is vital if some unidentified network 
1s to be associated with a particular model network 
contain in-g such a pointer* Figure 4-15 shows some of the 
satellites of MARftYS* 

Each type of satellite is associated wUh a type of c - 
note forming an open ended family. Thus 1n addition to 
negat ive- sate! 1 ite-pa ir c-notes* there are probably 
satell ite-pa ir c-notes, ma ybe-satel 1 ite -ua ir c-notes, must- 
satell ite~pa1r c-notes, must-not-satel 1 ite-pa ir c-notes and 
so on* 
4.4.3 Concept Modifications 

■ 

Frequently the members Of a lfnked pair both have 
pointers to closely related concepts. For example* If a 
brick in one scene fs linked to a cube in another, the 
situation is as shown in figure 4-16, This 1s very much like 
the po1nt#r*sa tell ite idea with A- KIND -OF replacing 




PI - MAYBE- SATELLITE 
F-2 = PROBABLY- SATELLITE 
E3 = NEGATIVE- SATELLITE 
P4 = KUST- BE- SATELLITE 
E5 = KtfST-NQT-lE- SATELLITE 



FIGURE 4-1J 



^^^^ LINKED >^™Ni 

o — o 




FIGURE 4-16 



C-NOTE 




A-KfNJ-tJF 



LEFT -BESTINAI ION RIGHT- DESTINATION 



FIGlffiE 4-17 



124 



MODIFICATION -OF * In any case, the description generator 
recognizes this and slrailar situations and again generates a 
group of c-note types. The first of these is the A-KIND-QF 
chain Illustrated by the above situation. This causes the c- 

note of figure 4-17* 

The a-k1nd-ef-chain c-note also includes situations in 
which one concept is related to another not directly* but 
rather through two or three A-KIMD-GF relations. Suppose* 
for example, a cube is 1 inVed with an object for which no 
fdent if icat ion can be made. There is Jtlll an a-Mnd-of- 
ehain c-note because cube is linked to the general concept, 
OBJECT, by a seguence of A-KIND-QF relations, [figure 4-18) 

Another kind of pcnular concept modification is the a- 
kind-of -nerge c-note< These a -k 1nd -of -nerge c-notes ocdir if 
there is no A- KINO -OF chain as described above, but each 
concept has a chain of A-K1ND-0F pointers to some third 
concept. For example, WEDGE and BRICK are both connected to 
the concept, OBJECT, byA-KIND-DF, (figure 4-19} 
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f, Learning and 'lodel Building 

5.1 L ea r n i n g 

In this dh4pt«r I discuss learning to recoqnize simple 
block configurations* Although this may seem 1 fke a very 
special kind of learning* I think the implications are far- 
ranging. 

It is possible to assuime extreme positions on the 
subject of learning. One person may think learning to do 
things is very c ompl ita ted » while learning to recognize 
things 1s comparatively staple* because one merely aquires 
templates or some such. Conversely* another oerson may think 
learning to do is sinple, but learning to recognize Involves 
deep Gestaltlst problems of synthesis or other imped tmenta* 

My opinion Is that learning by examples, learning by 
being told, learning by imitation* learning by reinforcement 
and other forms are much like one another, 

In the literature there is frequently an unstated 
assumtion that these various forms are fundamentally 
different, But I think the classical boundaries between the 
various kinds of learning will disappear* once superficially 
different kinds of learning are understood in terms of 
processes that construct and manipulate descriptions. No 
kind of learning need ce desperately complicated once the 
descriptive machinery is avail* tie, but all constitute 
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opaque, intractable processes without it. 

Then once the problem of using descriptions is 

thoroughly understcac, it will be oosslfcle to give meaningful 

thought to a deeper problem, that -of learn ir.c to use 

descriptions. It does not seem simple, but usinq this point 

of view,, it seems less than impossible* 

The notions of learn ing and teaching are broad and 
confused, Generally, people think of these things as 

occurring toqetber, so that whenever something learns, 
something else teaches. But somehow intuitive notions 
fall when it corce & to thinking about machines, 
Computers can now play tolerable and improving chess and 
do marvelous symbolic integrations. Vet while people 
freely use the word "teach" in describing what the 
programmers do, hardly anyone thinks of the machine as 
1 earn ing. 

The reason seems to he that the human programmer 
Has supplied so much detail that the machine is more a 
mimic than a thina with learning ability. The machine 
acquires its skill without ever really knowing what is 
going on or how 1t might improve without the laborious 
services of an Information processing surgeon- The 
unfortunate machine is in the pesitior of school pupils 
who know f^cts and perhaps memorize simple algorithms, 
but are never programmed to learn* The programs to be 
discussed here are an effort to show that a machine can 
do better and can learn in a realistic sense, given a 
chance. 

&*Z Descriptions and Models 

I want to irake a clear distinction between a description 
of a particular scene arid a model of a concept. A ircdel is 
like an ordinary description 1n that it carries information 
about the various parts of a configuration* But a model is 
more in that it exhibits and Indicates those relations and 
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properties that must and must not he in evidence In any 
example of the concept Involved. 

Suppose, for example, the description generating 
program* report the following facts 1n connection with the 
arch: in figure 5-1: 

1 . Object A is a brick. 

2. Object A is supported by B and C* 

Now suppose the description containing these facts were 
compared with the scene in figure 5-2 , where object ft is a 
wedge, and with the scene in figure 5-3, where object ft Me 5 
on the table. In both cases eotioarison could be made and 
differences appropriately noted, but the identification of 
cue or the other nf these new scenes as arches would be risky 
indeed because so far the machine knows only what one arch 
looks like without knowing what in that description is 
important I 

Humans, however, have no trouble Identifying the scene 
in figure 5-2 as an arch because they know that the exact 
shape of the top object in an arch 1s unimportant. Dm the 
other hand, no one fails to reject the scene 1n figure 5-3 
because the support relations of the arch are crucial. 
Consequently, 1t seems that a description must indicate which 
relations are mandatory and which are Inconsequential before 
that description qualifies as a model* This does not require 
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any descriptive apparatus not already on hand. One need only 
substitute emphatic forms like WU ST -BE- SUPPORTED -BY for basic 
pointers like SUPPORT ED -3Y or, in some eases, add new 
pointers* Discovering where and when to perform these 
operations can be somewhat involve^ however, and requires 
the bulk of this chapter for discussion* 
5,3 Examples, Kear -«i 1 sses, a fd Non -exann les 

5uppose it is desirable to train a machine: to reconni2e 
the letter A without restriction as to type size or font. 
The designer then has two sets of options; He must decide 
how his -machine is tn work; and he must decide what to show 
the machine. One idea is to show the machine vast numbers of 
As and hope that it will benefit from such an eaner fence by 
somehow noticing the features which a p near repeatedly* But 
this assumes that the frequently seen properties are 
essential ores, which can he a had rule. Moreover, there is 
little possibility for skillful teaching. There fs no 
obvious way the teacher could quickly convey a particular 
idea such as the notion that, the crossbar of an >\ is 
i-.-jfrt i "t , f;vrTi i- 1 " L-,l: iw.chor rea'-'zei it.. Fi'i.il" .y t <jcr 
sample-s generally only sunnest properties a candidate for 
natch should have -- ft is hard for them to indicate 
^'or Hidden properties. 

This is true because expert description programs 
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would be needlessly overburdened and would promote 
confusion if they were always to indicate all negative 
properties and those not observed. Therefore a 
property's consistent absence can be easily overlooked* 
If the description prooram does list dill properties In 
spite of Inefficiency, then many properties are 
statistically likely to be missing from a short training 
sequence. But the stat ist ic-ga therintj machinery would 
think Such properties should not be oresent In examples 
of the model, even though their absence was by 
c oinc id enc e. 

One might attempt to train basically the same machine to 

handle a whole repertoire of concepts by showing examples of 

each member. Thus 1f the repertoire were the alphabet, 

examples of all the letters would he shown* rather than just 

As, This shifts the question to l.'hich description does an 

unknown best match? It still avoids the more fundamental 

question* Nhat is It about each character that is essential 

and permits It to be recompiled? The machine new may use As 

and non-As, but the difficulties are only obscured, not 

c Ircumv entetL There remains no way to directly convey an 

idea t and there remains the fallacy that frequent anpearance 

means importance. The problem of Indicating what properties 

preclude identification w1*h a particular model is only 

tangential ly and occasionally addressed in that sumetines a 

property converts one description into another, as in the 

case of adding a forbidden midline crcsshar to a C, The 

machine does r:pt know a C- cannot have a crossbar; it only 

knows such a crossbar nates a figure nore like an E t 
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In my judgement t near misses are the really important 

examples 1n learning. In conveying th& idea of an arch, an 

arch certainly should be shown first. But then there should 

be some samples that are hot arc he s^ bjt do not miss beinq 

arches by much. Small differences permit the machine to 

localize some part of its current opinion about a concept for 

improvement . If one wants the machine to learn that the 

uprights of an arch cannot marry, one should shew it a scene 

that fails to be an arch only in this respect. If the 

machine is to know a C cannot have * crossbar^ it should see 

a character that fails to be a C only because of a crossbar* 

Such carefully selected near misses can sugaest to the 

machine the important dualities of a concept* can Indicate 

what properties dTe never found, and permit the teacher to 

convey particular ideas Quite directly. 

It is curious how little there is in the literature 
of machine learning about mechanisms that depend on good 
training sequences. This may be partly because previous 
schemes have been too inadequete to bear or even invite 
extensive exploration cf this centrally important tnoic, 
Perhaps there is also a feeling that creatfnci a training 
sequence Is too much like direct crooramm in n of the 
machine to Involve real learn inc. This is probably an 
exaggerated fear. r agree with those who hellsve that 
the' learning of children is better described by theories 
using; the notions of programming and sel f-programminn> 
rather than by theories advocating 1 the idea of self^ 
organisation. It is doubtful, for example, that a Child 
could develop much intelligence without the programming 
implicit in his instruction, guidance^ closely 
supervised activity, and general interaction with other 
humans. 
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5*4 Model Development 

The mathlne^s model building program starts with a 
description of some example of the concept to be learned. 
This description is itself the first model of the concept. 
Subsequent samples are either examples of the concept or near 
m1ss.es. These examples and near misses reveal weaknesses and 
lead to a newnodels. Section 5.5 shows 1n some detail how 
the comparison between the current, model and the description 
of a new sample produce a new model In the case where only 
one d if f erenc e Is found. 

One then has 4 seouence of more and more sophisticated 
models. See figure &-4, Frequently, several responses may 
appropriately address the comparison between the current 
model and a new sample, when this happens, branches occur in 
the model development sequence and It is convenient to talk 
about a tree of models. Figure 5-5 shows such a tree. 
Section 5*6 discusses how the alternative branches come 
about. The machine selects one branch at each point for 
further development. The meandering path leading from the 
top Of the tree down to the current model is called the main 
line. The main line changes course when a particular 
sequence of branch selections leads to untenable situations. 
Section 5,7 describes how and when this happens. 
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5,5 The Elementary Model Building Operations 

This section cc*is1ders the case in which the matching 
program finds only one difference between the current model 
and i new example or near miss. The table at the end of this 
section summarizes the results, 

5*5.1 The fi-kf rid -of -mer get Current tfodel and Example 

First consider the model and example in figure 5-6* 
Figure 5-7 shows the resulting comparison description* Only 
one difference is found: the object of the model points to 
BRICK while the object of the example points to WEDGE* But 
Since both BRICK and WEDGE relate by A-KIKD-OF to OBJECT, the 
a*kind -of -merge c-note occurs, Several explanations and 
companion responses are possfble. One 1s that the source of 
the c-note may In general point to either of the things 
pointed to by the A -KIND -OF pointer In the two scenes. Thus 
the object could he either a WEGGE or a 8UICK. Another 
possibility fs that the ft -KIND -OF pointers from the object do 
not matter at all and can be dropped from the model. Still 
another option and the one preferred by the program i% that 
tie object may be any member of some class in which both 
WEDGE and BRICK are represented. In the example one such 
class is simply the concept OBJECT and has already been 
located as the intersection of A-KJN0*QF paths. The program 
responds by replacing the pointer in the comparison network 
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that points to the a-kind-of-merge e-note by an A-KIND-GF 
pointer to the Intersection op merge concept* In this ea se 
an fl-KIHD-OF pointer 1s installed between the e-note origin 
and the concept OBJECT* Then the altered comparison network 
is the new model shown in figure 5*8,, 

T he primary response I have selected fcr the 
machine reoresents a moderate stand with respect to a 
rather serious Induction problen* I coult* have been 
more conservative and used an A-KIHB-OF pointer to a 
concept representing the ordinary fl^ of BRICK and WEDGE, 
On the other hand, I could have been far more radical by 
pointing the A-kIMO-QF pointer to THING, the universal 
class, y\y actual choice of pointing to the node 
indicated by the merge concept to he the Intersection of 
A-KTNO-OF chains seems more flexible than either of the 
ext re-ems. 

Sfnce the merge concept is itself defined in terms 
of the networks content, the generalizations ma-d e will 
change as the net grows. Cut since the aporoprfate 
response should after all depend on the universe the 
machine is operating, in, the generalization changes are 
likely to be improvements. And in any case this 
commitment 1s only one among; many. 

5.5.2 The Su pp 1 ementary-point er C-note 

Now suppose scene 1 in figure 5-9 represents the current 

model while scene 2 contributes as a near miss. The matching 

routine soon discovers that scene 1 produces a SU PPORTEfl -BY 

relation between the two objects whereas scene 2 does not * A 

supplementary-pointer e-note results* Of course the 

implication is that the concept studied reouires the two 

objects to stand together under the support relation. 

Conseau ently t when such a su pplementary-nointer c-note turns 
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up, It transforms to the emphatic MUST version of the pointer 
involved. Thus the new model is the one In figure 5-10, 

Of course the supplementary pointer can turn up in the 
near* mfss is well as In the current modeli Suppose scene 1 
in figure 5-9 is the near miss instead of the current mad el . 
One concludes A cannot be on B, The supplementary-pointer c- 
FiDte now indicates a relation tnat apparently cannot hold, 
A ppropriatelVi the KJST-flQT version of the su pn 1 ementary 
pointer Is substituted in and the new net appears as in 
f 1 gu r e 5-11. 
5,5,3 The Mu st- sa tel 1 ite-pa ir C-note 

Frequently comparison between the current model and a 
new sample displays c-netes that do not reveal any new 
feature, but rather result because of previous refinements fn 
the model. Suppose, fpr example) that the current model has 
a MUST -MARRY pointer in a given location, while the sample 
has a MARRY S pointer. Now clearly the HARRYS pointer is 
appropriate in the description and the mu st-sat el 1 tte-pa 1r c- 
note consequent to (matching It with MUST -MARRY should be 
replaced a^rain by MUST -MARTI V. Thus the emphatic form in a 
mu st -satell ite-pa ir situation is retained and not interfered 
with by refinement operations attempted subsequent to its 
formation, 
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5,5.4 The A-klnd-of-merne: Current Model and Near ^ t ss 

Sometimes a e-note offers two *>r more nearly equal 
explanations. Consider the super simple current model and 
near miss in figure &-12. The c-ncte Is an a- ktnd-of-nerne 
announcing that the current isodel points with HA5-PP0PEP.Tr-0F 
to STANDING, the near mfss tc LYING, and both LYING and 
STANDING have A- KIND -OF paths to ORIENTATIONS, Wow the near 
nlss may fail either because it is lying or because it is not 
standing. Responding to these exp lanat ions , the model 
builder night replace the a « kind -of -merge c-note by a MUST- 
NOT-HAVE-PRDPERTY-OF pointer to LYING Or by a MUST -HAVE* 
PRQPESTY-DF pointer to STANDING, Since mast concepts humans 
discuss are defined in terms of properties rathor than anti- 
properties, the MUST verslcn is considered more likely. 
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ACTION OF CONCEPT GENERATOR; EXAMPLE CASE 
onote type pointer involved rejpcnse 

a-kind-of-chain 



point to intersection 
^Ith model's pointer 



a-kind-of-merge 



K point to intersection 
with models pointer 
Z. drop model '5 pointer 



negative -s ate] lite 
pair 



drop model 's pointer 



must-be -satellite 



retain model's pointer 



must- not- be 
satellite pair 



contradiction 



s uppl eflien ta ry - po i n tc r 
or exit 



negative-satellite or 
fundamental pointer 
In the model 



drop model 's pointer 



negative-satellite or 
fundamental pointer 
in the example 



ignore 



must-be-satellite 



contradiction 



must-not-b-e-satellite retain model's pointer 
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TABLE {contj 
ACTION OF CONCEPT GENERATOR; NEAR MISS CASE 



c-note type 



pointer Involved 



response 



a- kind- of- chain 



if model "s node is at 
the end of the chafn 
add must -not -be satellite 
f f near miss ' node is 
at the end of the chain* 
use irust-be satellite 
to model 's node 



a-kiind-of -merge 



1, replace model 's pointer 
by its must-be satellite 

2. replace model 's pointer 
hy must-not-be satellite 
of near miss 1 pointer 



negative-satellite 
pair 



replace model's, pointer by 
Its must-be Satellite 



must-not-be-exbension 
pair 



retalni model's pointer 



s u ppl ementary 'po i nter 



fundamental pointer 
in the model 



replace pointer with Its 
must-be satellite 



fundamental pointer 
in the near miss 

negative- satellite 
in the model 

negative-satellite 
fn the near miss, 



insert pointer into the 
using must -not -be satellite 

replace pointer with Us 
must-not-be satellite 

insert pointer into model 

using must-be satellite 
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5.G Multiple C -notes 

Ccmpar i sons yield f n a sinnle c-notes are rare, More 
often, the model builder must flake sense out of a whole group 
of c-notes 4 If the comparison Involves a near miss, any one 
of the c-notes might be the key to proper nodel refinement* 
Moreover, many cf the c-notes have- alternative 
interpretations that make further demand's on executive 
expertise* 

The model builder must therefore consider the c -notes 
and all the possible interpretations of each. Then it must 
produce the set of hypotheses that form the nodel tree's 
branches. These in turn must be ranked so that the test r*a y 
be pursued first* 

The case of ref ioement through *r example is simpler 
than throunh near misses* Since none of the observed 
differences are sufficient to remove the example- from the 
class, it Is assumed that all of the differences found set in 
concert to loosen the definition embodied in the model, 
Consequently each c-note can be transformed indRoendentl y and 
a n evi model tienerated fcy their comblnod action. There is no 
pr obi era of dec Id inn if one difference is more important than 
another H 

Consequently, if all the c-notes had but one 
interpretation, only one new branch *>ould be Generated, The 
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a- kind -of -inert;*! c-note has three possible interpretations, 
however, and if one such c-note occurs, it is only reasonable 
to create three branches Instead of just one. The action on 
the other c-noles is the sane for all three branches, 

:Jear misses cause more severe problems. If two 
differences are found* cither of them nay be sufficient to 
cause the sample to be a rear miss, while the other 
difference may be equally sufficient or merely irrelevant. 
If the differences have multiple Interpretations or more tran 
two differences occur, the number of possibilities explodes 
and the machine cannot work simply by generating an 
alternative for each possibility. 

The model builder clearly must decide which 
interpretaticn of which differences *re most likely to cause 

the near miss. 

The machine first forms two lilts: a primary list and a 
secondary list* Each c-note eventually ends up In one list 

or the other. 

Now some c-notes can never mate the primary list because 
they arc- cf tnerr.se- Ives insufficient to explain why a nivert 
sample is a near miss. All of these no immediately to the 
secondary c-note list. One example is the situation In which 
a pointer in the near miss corresponds in the current model 
to an instance of the MUST-SATELLITE version of the pointer. 
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A must-satell it e*pair c-note results but certainly Is no 
orcunds for excluding the near miss frGfl the class since the 
required pointer is in fact present* Some other explana t i en 
mu St be foil rd . 

The next and most obvious way to sort d iff erei'K: f-s is !■:■■ 
level. This assumes only that the differences nearer the 
origin of the comparison description are the nore inportant* 
This certainly is ft reasonable heuristic since a missinn 
group of blocks generally impresses a human as befnfl no re 
important than a shape change* which in turn dwarfs a minor 
blemish. Consequently, the program determines the depth of 
the remaining c-notes which are nearest the origin of the 
comparison descr lot Ion. AH those e*.nc<. idatos found at 
greater depth are relegated to the secondary list. 

The primary c-note list allows quick formation of little 
theories about why the near miss misses and ivhat to do as a 
consequence. These theories are called hypotheses* A 
complete hypothesis specifies one e-note as the scle cause of 
the Piss and it further specifies which interpretation of 
that c-note is assumed. Consequently there is a hypothesis 
for each interpretation of each c-note on the primary list* 

The c-note specified as crucial by a hypothesis is 
transformed as if it were the only c-note. The other c- 
notes* both on th* secondary and nr inary lists* trn assumed 
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by the hypothesis to be tnsu ff ic ieri t cause for the near m "5 ss, 
Co n sequ entl y as a new ncdel is f emulated accord inn to the 
hypothesis* all of the c-notes but one are treated exactly as 
if the near Piss were not a miss at al 1J 

5o far a single c-note is assumed to be the exclusive 
cause df the Ms Si Here all possible c run H na t ions considered 
as well, not only would the branching increase enormously, 
but the rankinn. of those branches i/ould be difficult. Rather 
than face this, 1 have decided that only one special 
combination of two c-notes is ever permitted to form a 
hypothe si s„ 

In this I have exercised what one Plight call the 
first heuristic of science: Benin with the linear 
model* the one that assumes all thinos act 
independently; then consider interactions as necessary, 
I next discuss a particular case in which It does seem 
necessary to consider tit- joint action of two 
differences, It would be unreasonable* however, to try 
for a general method fnr handling multifile differences. 
In science as a whole, each particular method for 
treating interacting effects is u sally a major problem 
in Itself and over-am hi tinus search for completely 
■general methods is of low utility when premature* 

Further justification for my appprnch lies in 
certain observations of Piaget's that Indicate that 
children seem to pa y sharp attention to only a single 
feature at any one time [3]. In comparfnn volu/fies f for 
exam pie t they use mainly height. Vet* in spite of using 
what appear to be linear comparisons, these same 
children can learn physical concepts with a talent far 
in excess of my goal for this thesis* 

Hypotheses based on two c ontr f but in-ci c-nntcs are added 

to the hypothesis list only when two c-notes with nearly 
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iu c-r. t leal c c scr i p-_ *" ens o:cur. C en s idf?v ^ i \u. rx i-]'l, Sv.\C.f. 
exactly the 5ame thing characterizes both blocks in the near 
nnfsSi there is no particular reason to suppose that one 
difference should be singled out. Consequently a third 
hypothesis is forrned a namely th?t both differences act 
cooperatively* This additional hypothesis- takes precedence 
over the two hypotheses that consider the differences 
separately. It seens heur i st ica 1 1 y sound that coincidences 
are significant. The machine creates new models nith such 
hypotheses by transforming both of the specified t-notes in 
the miss-explanation node* 
5*7 Contradictions and UacHng Up 

Cy now ana may wander why the program should deal with 
alternatives to tne r.raln 1 1ne of model development at all. 
To be sure, n;a* inum likelihood a s sunn t ions nay be wrong, but 
then now could the machine ever know when such a decision is 
an error? The answer is that th9 main line a s suroot Ions may 
lead to contradiction crises which in turn cause the ™d<*l 
bufldtng program to retreat up the tree and attempt t^odel 
development along other branches. 

Consider again the very simple situation presented tn 
figure 5-14+ The current model and the near nlss 
combination generate an a-k ind-cf -nerge c-note for which the 
priority interpretation Is that examples of the concept irust 
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be standing* T fie alternative, that e*spples roust not be 
lying, causes a side branch in the rondel development tree, 
But suppose one really wants the concept to exclude lying but 
not Insist on stand trig. Showing the machine the example in 
figure 5-15 does the job. The tilted brick certainly Is not 
standing and its description has no KAS-P*CPERTY-OF pointer 
to STANDING, Vet the current model has a '1U ST - HAV E -PROPERTY- 
OF pointer to STANCMG. This is a contradictory situation, 

Vfhen contradictory situations occur, the program assumes 
It has made an incorrect choice somewhere*, closes the branch 
to further explore tion, and tacts up one level to select 
another a Her native if any are available there, tf no 
alternatives are available, the program tacks up still more 
levels until either an unexplored alternative is found > or 
the top level is reached- If the top level is reached with 
no other options found* the program succumbs and admits 
failure* More often an acceptable unexplored alternative is 
soon found and an effort is nude to extend the model tree 
down that branch. Of course* the first alternative a 
contradiction causes to be explored may itself lead to 
contrad lotion* Sack up then starts from the new 
contradiction and proceeds as before* 

In the case at hand* an alternative Is found and the 
must -not- be -lying interpretation of the comparison between 
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the scenes 1n figure 5-14 leads to a new intermediate model. 
This In turn Is refined by the scene of figure 5-15 nhich 
originally caused the ccntrad let ton on the former main line. 
Ho contradiction occurs on the new path because the 1 -1L15T-NQT- 
HAvE-PROPERTr-DF - LYIHfi combination of the intermediate 
model has nothing to clash with fn the example. Indeed the 
new example lends no new information to model development 
along this path f the model being the sane before and after 
comparison. The new example served solely to terminate 
development of an improper path in the model development 
tree , 
5.3 Other Backing Up Possiti 1 it 1e s 

Many possible refinements to the elementary hack in a up 
procedure invite attention. For one thing there are other 
reasons why the learning program plight want to back up. In 
addition to the situation of direct contradiction* attention 
should move back up the model tree if there are so many 
differences between the current model and the sample that 
hopeless confusion is suggested, The cause of such confusion 
is likely to lie in the selection of a wrong branch at some 
higher point In the model tree* Similarly retreat is in 
order if the program is forced to prooose an unlikely 
explanation to account for observed differences* 

Right now the learning program backs up level hy level* 
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blindly exploring all possible paths from one branch point 
before bucking up to the next higher hranch point. It would 
be better If attention could nove directly to the nolnt In 
the tree where the problem began. There a better alternative 
could be elected a net learning tnuld more likely precede In an 

orderly my. 

Certainly the selection of the appropriate point Is easy 
in the case of direct contradiction, As explained before* 
these situations occur when some relation 1s found to be 
essential at scire point In nodel development only to be 
absent froP sane subsequent example of the concept* The 
crucial point is the place where the decision was made that 
the relation was essential. This is the point where 
attention should go and an alternative explanation should be 
sous ht . 
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6 Some Generated Concepts 
6.1 Physical Models and Functional Models 

Ir this chapter I explore some of the properties of the 
model generator through a series of examples* In the course 
of this discussion, words like house, archi and tent occur 
frequently as they are convenient names for the ideas the 
machine assimilates, Re cautioned, however, to avoid 
thinking of thesE entitles in terms of functional 
definitions* To a human, an arch nay be something to walk 
through, as veil as art appropriate alignment of bricks* And 
certainly, a flat rock serves as a table to a hungry person, 
Although far removed from the Image the word table usually 
calls to mind. 

But the machine does not yet kn oij an yt hire of waHing f 
residing, or eating, so the programs discussed here handle 
enly some of the physical aspects of these human notions- 
There Is nothing mystical Stout this. There 1s no 
Inherent obstacle forbidding the machine to enjoy 
functional understanding. it is a matter of 
generalizing the machine's descriptive ability to acts 
and properties renu ired by those acts. Then chains of 
pointers can link TABLE to FOOD as well as to the 
physical linage of a table, and then the machine will be 
perfectly happy to draw up its chair to a flat rock with 
the human, given chat there is something on that table 
which It wishes to eat. 
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:,2 The House 

Figure E-l illustrates what house means here* Basically 
the scene is just one wedge on top of one brick. But lacking 

human experience, this one picture is insufficient to convey 
much of the notion house to the machine. The model builder 
must he used, and it must be pernitted to observe other 

samp 1 6Si 

Suppose the model builder starts with the scene in 
figure 6-1, Then it s de scr Ipt Ion generation apparatus 
contributes the network which serves as the first unrefined, 
... ■'-.ci-bc-r is'-*:-.: io.: c".. [fiu'-ri? V.- 1 .;) i ■■■ r-.irrnv- fn ncr."* in 
figure 6-2 f a near miss, is the next sample, Its net is that 
shown in figure 6-6. The onl y d Iff erence is the 
supplementary pointer SUPPORTtD-'EY. Glancing at the table of 
section b.b, it is clear that the overall result is 
conversion of the SUPPORTED -BY pointer in the old model to 
S1UST-BE-SUPP0RTEEUBY in the new model. Thus the new model is 
that of figure 6-7, Figure 6-fi shows the current model 
development tree. 

iuch Is yet to be learned* For one thirty the top 
object certainly must he a lYedne. Shewinci the machine the 
near miss of figure 6-3 conveys this point immediately. 
Similarly the near miss of figure 6-4 makes the brick 
property of the bottom object mandatory. Cut notice that 
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both of these steps cause bifurcation of the model tree. The 

reason is that the iaehii*e cannot be completely sure the Mss 
occurs because the ol d property is l?s* or becau &e the n ev 1 
property is added,. The prcoran nrefers the old -prnnerty-i 5- 
Tost theory and moves down t he correspond 1 no branch unless 
contradicted. In both of these situations, the preferred 
tneory is correct resulting 1n the final model and tree shoi-m 
in f inure 6-9 and f inure 6*10* 

6.3 The Pedestal 

Dev elopmcrt of a neclestat model proceeds, much as does 
the house with only two essential differences. Firsts the 
top object must he a brick rather than a wedoe. Second, the 
upper object nust not rcarry the lower* The sr, pne In 
flours- G-11 yields the starting model. Fioure 6-12 forces 
the top object to ^e a bricfe while f inure P-13 forces the 
bottom object to be a brick as well, Flflure 6-14 emphasizes 
support. And finally, f inure 6-1 S forbids the HARRYS 
relat ion . 

f . 4 T he T en t 

Think of the tent as two vied pes, siarrylnn each other* 
As such it illustrates the hand If no of two similar 
differences simultaneously* 

Suppose the base model is the description of the scene 
in f inure 6 -IE- and the first sample is the near miss In 



ONE-PAEI-IS 




KU5T -BE- SUFPORSEU- £¥ 



HUS3VBE-A- 
KIND-OF 



FIGUHE G-9 



MODEL 1 



HflUEL 2 



MODEL 3 



MQDEL 4 




FIGURE 6-10 




FIGURE 6-11: PEDESTAL 



MEAR MISS 




FIGURE 6-12 



NEAR rriss 




^^ 


^^\ 






^ 




A 


"> 


FIGURE 6-13 



near niss 



vi 



^^ 



FIGURE S-H 



near niss 






-^ 


x2 


1 






^ 




FIG 


URE 6-r 


5 






NEAR PIISS 






FIGURE 6-17 



NEAR HISS 




FIGURE 6-18 



If7 



fiaure 6-17+ Tvo a - kino -Df-nerqe c-note* result* one from 
each of the two cbjects because they are brie Its riot wedges* 
Since they differ only In. source, the hypothesis that both 
ACt together is the priority one, which leads to the result 
in figure 6-19, Now this fs conp 1 enented by the near c i s e in 
ffoure 6*18 which Informs the machine of the Importance of 
the h '1ARRYS relation. A c? a in [iual c-notes announce the toss of 
a pair of ffARRVS pointers and twin M U £T - f t A R ft Tf raters are 
installed, [figure 5-20) 
E, 5 The Arch. 

The arch involves a mixture of the elements seen in the 
previous examples* Gecause of the wider variety of 
differences encount ered , it produces a bushy tree and a 
challenge to routines that select priority hypotheses* 

The scene i ti figure 6-Zl Forms the First model,, 
Combining this with the scene in fioure E-22 f the machine 
deduces that the ;*APKtS re 1 at lens bet men the top and the 
supports are not crucial + 

Next the near miss of fioure 6-23 Indicates that the 
support relations are crucial, ^qain* both new MUST -BE - 
Sll P PORT ED -BY pointers are hanuled jointly, and are installed 
at once. 

The machine learns perhaps the most Important fact frnr.i 
the near miss in figure 6-24 + Here the two supports touch* 
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supplying tuo "1ARRY5 pointers to the d escript icn + This 

cannot be allowed, responding, the machine Inserts MLI ST-NOT- 

M£RRY pointers between, the two supports in the model* 

Some my think that tn asserting the MUST -'JUT -HARRY 
relations, the machine overlooks what they consider the 
real principle, that of a hole or passage* But for a 
child building with blocks, to have a hole and to have 
two non-touching supports are very nearly the same 1dea + 
Conse qu ent l y the machine's oplpion seems adequate for 
the moment. Eh per tments such as these may help to 
expose exactly vihat kinds of network relations are 
adequate for a model of human thin kino, from infant to 
adult. 

Finally, the top object Is not necessarily a brick. The 
sample In figure 6-25 teaches the machine that anything In 
the class OBJECT will do, since OBJECT lfes but one step 
removed hy an A-KtNU-Of pointer from both WEPfiE and ERICK, 
6*6 The Vfeiige 

The capabilities of the model builder certainly extend, 
beyond the level of object configurations,, whose descriptions 
allow the machine to learn about scenes, here the 
development of the wedge nodel Illustrates the point. 

Given the wedge in figure fi-26, the description 
generated is that of figure 6-27 * 

■■lextj comparison with a brick establishes a MU 5T-BE -A- 
KIHD-OF pointer to TRIANGLE, (figure 6-?S) 

&ut now suppose the parti y occluded object in 
figure fc-2 9 Is compared, first with the current model of wedge 
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and then with a description Cf 4 brick, fts figure 6-3C 
illustrates, the sur pr 1 sln-ci result 1s that both comparisons 
ha*e nearly the same descriptions. In both cases two 
r&ctanoles arc matched and a third side left unmatched. [rt 
one case the unmatched side Is another rectancle, and in the 
other N It is a triangle* But there is a& yet no ^ay to 
prefer one match above the other! 

The problem Is a lfttle Invoked, ^o severe mismatch is 
evident to the difference description evaluation program 
because the MU ST-EE -A-KIND-OF pointer is anchored to a node 
that 1s not matched. The model as it stands asserts firmly 
that if three sides are seen, one of them must be a triangle, 
but it does not assert that such a third side must be 
present, 

T - : & may k&if zc. .»: ■.. -_u at first, but the probleir is 
really the machine's lack of experience. So far only two 
configurations have contributed to the model. Consider tne 
result of refining the model with the scene in figure 6-29 
which 1s causing the trouble. It 1s a near miss. But the 
difference between it and the current mode} is an exit c-n-&t& 
to the triangular side. The model builder perceives the need 
for an Emphatic pointer and OH E- PART -MUST -BE 1s inserted* 
(figure 6-31 ) 

Kdw if this model is compared again with the scenE that 
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just refined the model, the one in figure 6-29, the result is 
an exit c-note bearing an emphatic pointer, OnE- PART -MUST -BE* 

Such a c-note strongly suqgcsts bad match to the evaluation 
program, and the apparent inadequacy disappears. 

The scene in figure 6-32 establishes a final ref In ement . 
This wedge shows only two sides, After a bit of thinking, 
the program decides cine of the two rectangular tides Is 
optional and produces its final, model, [figure 6-33} 
6.7 The Composite Column 

When a concept Involves groups of objects, the nicdfl 
generation problen really is no nore difficult* It just 
becomes a matter of concentration on relationships between 
the typical rsernbers of the groups studied* 

Consider the notion of the composite column, hereafter 
referred to simply as the column, Figure £-34 shows such a 
column and figure 6-40 shows part of the correspond Inn 
descriptive network, This tic scr let f on is err a dually 
transformed into a reasonable model in the following way: 

Figure 6-3 5, iiti its bricks askeiv but otherwise the 
same, introduces the MU5T-MARGY pointer, Moure 6.-36* made 
of wedges instead of bricks, relaxes the inclination tovarrt 
bricks* And figure 6-37 causes rep lac ewent of Sil PPftftTFfc-BY 
by MUST -BE-SU PPftR TED *&Y. 

Next, figure 6*3-8 contributes the nost inportant part of 
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the mode!, the part that demands a group* The scene in 
figure 6-33 consists of only two objects and is not grouped 
because the grouping program requires a minimum of three 
objects* The resulting c-notes reflect the mandatory need 
for a, group by way of a Ofl E*PART-HU ST -BE pointer to the node 
representing the group. 

Finally, figure 6-39 generalizes the model in an 
Important way because Its typ ical -neater node differs from 
the model's principally because a different number of objects 
is indicated. In one case the NUMBER -OF -HEM HEP. S pointer 
points to 3; 1n the other, 4, But since both 3 and 4 are 
integers and have A -KIND -OF pointers to INTEGER, when the 
conparlson between the two is made aft 9 - kind -of-merge c-flote 
results. The next model consequently has a MUM BER-OF -MERCERS 
pointer to INTEGER, rather than 3. Nov.' columns may have any 
number of objects greater than two* 

Figure 6-41 displays the overall result* 
3*8 The Arcade 

The problem of learning about the arcade shown in 
figure 6-42 adds another interesting dimension to the model 
generation problem* Nothing new is needed in developing a 
sequence of models for the arcade, nith one Important 
exc ept ion: 

The arcade is a conglomeration of su bstructures,, rather 
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than of s1n.pl* objects. As such the arc.d. development show, 
now the model builder can appeal to thlnqs already learned In 
the process of understanding more complex structures- 

In building a description of an arcade, the description 
programs Identify arches using the previously assimilated 
arch model. This 1 cad* to the description partially shot* 1« 
figure 6-43, Then subsequent {tuples, shown in finure E-44, 
figure 6-45, and finure 6-46 Inform the machine that there 
must be a croup, that the group elements nust b* arches, and 
that the relation mu st be UI-FR 0117-01= . 

The deduction thet one arch 1 i In front of another 
Involves methods less explored and less scund than 
techniques previously described for dea 1nq with 
KSl! dual Ejects. Indeed this 1s a vlfj.n f ^ of 
inquiry that ] have thought about only enounr to « . 
prowls *Mch can handle these few otanp es ; It Is no1 
clear for example, tf eacn structure Hill require US 
own set of heuristics for determining Inter ^roup 

htons, or if oeneral pr inc ipl es can be enure ated. 
So far my prlnitfve programs assume only the following 
ru! 65: 

i if there is a cnaln of support relations 
between every object in structure A to sums object 
in structure H t the G can be said to support A. 

Z t if soae object in A 1s in front of some ° b ;* c *; 
In B, but not vice versa* then A can be said to be 
in front of B. 

6*9 The Table 

The table is much like previous examples except that 
grouping is done an tne basis of object form and function, 
rather than relation chains. 
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Study the table in figure 6-47 and the description 1n 
figure 6-48, The essential features of the table are 
introduced" by the following sequence of steps: 

First the tabic should have bricks for legs. This idea 
is easily conveyed by the non-table of figure 6-49. 
Moreover, this conception of tabic excludes structures such 
35 that in figure 6-50, a fact which is handily incorporated 
through a HU ST -KOT -MARRY pointer* Next, since the non-table 
in figure G-51 has only two supports, no grouping occurs, 
which leads to Insistence on a group in the neat model 
refinement. This entirely parallels the process by which 
the ealuin was found to involve a a roup* Finally* the scene 
In figure 6~52 leads to replacement of the SUP PORTER -BY 
pointer by HUST-BE-SUPP0RTED-3Y. Figure 6-53 shows the last 
model in this deve lopmeflt . 
6* 10 The Arch in Depth 

So far the illustrations have shown networks only to 
that depth appropriate for understanding. Figure 6-54 shows 
the model for the arch in somewhat fuller bloom and better 
indicates the breadth of the information available to 
pro ar an s that use the node!. 
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0.11 Directions for ircprov emen t 

Experiments with the concept generator titillate rather 
than satisfy, for each success suggests new Ideas to be 
explored. As It stands, the concept generator Is a healthy 
baby but not a contributing adult. The possibilities are 
stagger ing f however, 

fine nay to intir^ve nnt'el feneration abilities is self- 
evident; the system improves as the ability to describe 
improves, The heuristic determination of support and 
occlusion ann the r« St can he improved t and more important* 
other sd far neglected relations and poverties lie 
unexplored. Description programs should, and can be taunht 
to differentiate between cubes, hricks T hoards, and sheets* 
They should knov in yen era 1 when the terns lylno and standing 
are meaningful* They should bo able to relate structures 
more carefully. They should present alternative descriptions 
if situations are pla inl y ambf guoms. They should knotr about 
color ar.d texture. They should knov; about holes, 

Eiu t as description becomes Letter^ the burden on the 
model builder becomes greater, for the proliferation of 
properties and relations means that the number of c-notes 
multiplies, thereby complicating all of the d&dslon 
processes. To cope, ft will probably be necessary to 
institute further techniques for locating the centrally 
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imp or ta rut c-notes* 

For one thinu, t-notes involving certain pointers might 
rate priority attention. These might* for example, be 
pointers that frequently placed central roles In the 
development of other models In the past* Additionally, soma 
pointer s m ight be tentatively converted to MUST -BE versions 
by virtue of frequent occurrence 1n the samples ur=der 
consideration* Previous o bj ec t i en s to this Idea still stand , 
but done with care, with the assumptions behind the action 
somehow indicated* some good might cone from this. 

If the machine could ask the teacher questions, it would 
open up another powerful kind of finesse. In situation* 
becoming precariously ambiguous, the mod#l builder would ask 
directly if some relation is Important, or perhaps display 
several Imagined scenes to the teacher, reeuestlng a 
statement about which are in the c Ta s s. to he. learned. 

Finally, and perhaps crucial, some confrontation needs 
to be mar:e with function* What to do, however, is unclear. 
In dealing vith the table, the rlevelo pment was already 
strained. It is not really adequate to think of legs solely 
as standing bricSts* and generalization to the class of all 
objects seems specious. So far all models and 
identifications deal only with descriptions of concepts' 
part Si but this Is not adequate to handle the notion of the 
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leg. What is nfi'tun! :j the id eb that sonethlnq i s a leg, not 

because of what Its parts are and how they arc related, but 

rather because that something relates to something else in a 

particular way. namely through the SJPPORTED*BY relation. 

Given the ability to think this way, programs could Identify 

round legs, square l£gs» or legs made of several parts, 

Functional use would serve as a very powerful Interpretive 

and grouping tool adding immeasurably to the limited 

understanding now a tta ined r 

I emphasize that no fundamental barrier prevents 
programs from thinking functions H y. Indeed some 
possibly useful programs already exist. As work 
progresses* further analysis can be attempted and 
identification can be expanded to include the 
relationships that suggest function, At first work 
should concentrate on things that art defined In terns 
of currently observed use t rather than things that are 
defined in terms ef conjectured! potential use* A table 
leg, for example, is a leg because It currently is 
observed to support something. The sane object would he 
far less likely to be Identified hy a human as a 1 en 
were it seen separated from any table* There is little 
possibility of identifying something as a table let? on 
the basis of potential use unless a leg is specifically 
searched for. 
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7 Identification 
7*1 Patching and Mentlf icat 1on Alternatives 

Once there are programs that describe scares, cn-n^rs 
description net work s, and build models, one may go on to 
using these programs as elements 1n a variety of other goal- 
oriented program. The prob/l em-solv ing programs described In 
this chapter have the foil owing kind of responsibilities: 

To see 1f two scenes are Identical. 

To contrast two scenes and report the differences 
between then; , roughly 1n order of Importance, This 
supplies information that may prove useful to programs 
that use a mechanical hand and arm to build copies of 
5C en e 5 , 

To compare some scene with a list of mo-dels and report 
the most acceptable match. This is the identification 
problem fn its simplest form. 

To identify sane particular subset of the objects In a 
scene, This is not the same as Identifying an entire 
scene because important properties may be hidden and 
because context may make some ident if icat 1cns more 
probabl e than others* 

To find instances of some particular model in a, scene* 
It is frequently the case that the presence of some 
configuration can he confirmed even though it would not 
be found in the ordinary course of scene description. 
This requires the ability to dfscern groups with the 
required properties in spite of a shroud of irrelevant 
and distracting information, It is not unlike the 
prc-blc^ of finding- the bunny on a Playboy cover* 
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7,2 Exact Match 

If two scents are identical t then the networks 
describing those scenes must be isomorphic. The nod^s of the 
two networ ks must relate with each ether in the same ways, 
a rid the nodes, mu St rotate to general concepts such as GIMCK 
and STANDING In the same ways* C en se gu en 1 1 y w con paring two 
sue Ji networks produces a simp If: kind of comparison 
description. There is a skeleton, which indicates hot: the 
parts of the scenes interrelate,, and there i & a group of 
intersection c-notOS that describe how the parts of the scene 
am anchored to the general store of concepts. None of the 
other types of c-notes appear because identical scenes cannot 
produce two networks with the noce &sary a berrat ion s of form* 

Conversely, if comparison of two networks results in 
intersection c-notes onl y t then the parent scenes must be 
identical in the sense that the de script for a en era ting 
mechanisms employed produce exactly matching networks- There 
can be variation p but notniio so great a s to vary the action 
of the description generator. The scenes in fitrure 7-1 are 
idciiticil *i T -h respect tc tne rlR^criptfve power cf ny 
programs because in both cases the relations observed are 
LEFT -OF and RIGHT -OF. More capable prograns micht comnlain. 
that FAR-TG-THE-LEFT»nF arid FAR-TO-THE-R IGliT -GF hold in one 
scene, while only LEFT-UF and RiGHT-OF hold in the other. 
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The scenes are clearly not Identical with respect to a 
program with such a capability, 

One generalization adds a decree of flexibility to this 
procedure that humans seem to exercise. The kind of question 
to be answered is not of the s Imp 1 e form "Are two given 
scenes identical?" but rather, "Are two given scenes 
identical with resect to a certain degree of detail?" 

An approach to this new problem Is to modify the 
Intersection-only criterion. Instead of requiring all c- 
notes to bo intersection c-notes, one requires that all c- 
notes be Intersection c-notes at or above a certain level, 
Thus if level one is specified, then the scenes in figure 7-2 
are indeed identical because the comparison description shows 
nothing hut intersection c-notes at or above that level, 
(figure 7-3} Cut if level two is also of concern, then the 
scenes &re not identical because an a- klnriJ-nf-mern* 1 r: -rot e 
describing the difference in shape between face C and; face C 1 

appears on that level. 

This use of level plainly defines a concrete substitute 
for the otherwise vague notion of deoree of detail* One 
simply says two scenes either are or are not identical down 
to some particular level and the existence of non- 
inter sect ion type c -notes beyond that level fs not of 
concern. 
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7,3 J 1 gr e s s 1 on 

The exact natch detector i"5 a major part of a curiously 
simple program that checks ftir ?. certain kind of left -right 
symmetry* The method is as follows: 

li Copy the description of the scene exactly* 

2. Convert ill LEFT-flF pointers in the copy to RIGHT -Of* 
and all RlflKT-OF pointers to LEFT-OF. 

3, Compare the original description, against the modified 
copy* If the natch Is exact! the scene fs symmetric* 

This is, of course, an abstraction of the familiar 
condition for y*ax1s symmetry In the mathematical sense, 
whereby symmetry fs confirmed if and only if for every point 
in the scene, (x»y)„ the point (-x»y} is also in the scene* 
St. itching LEFT -OF and RIGKT-OF pointers is the analogue of X- 
coordiflate negation and network matching corresponds to a 
check for invariance. 

To see how this works,, consider the scene 1n figure 7-4. 
The center object, ft, is flanked by & on the left and by C on 
the right* Figure 7-5 shows the resulting dfiscrigtlon. 
There are nodes correspond ing to objects A, B, end C, and 
there are LEFT*0F and R IGHT -;" 117 pointers indicating their 
relationships. 

Figure 7-6 shows the copy of the network with the LEFT- 
OF and RIGHT -OF pointers switched!* Notice that the original 
network and the copy are identical, ffode A matches with B 1 , 
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t with C'„ and B with A'. Since there are no differences, 
the machine concludes the scene 1s in fact symmetric* 

Of course, a genera Illation of symmetry is possible* 
just as generalization of identity is. The machine need only 
perform the same enpyi^c aTl< \ p x c.hanoe operations and then 
check for non- inter sec tiun c -notes onty to sone particular 
depth. Thus the scene in figure 7-7 is symmetric to depth 
one because the groups of objects are symmetrically placed. 
It is not symmetric to depth two, hcvever* because the 
placement of objects within the groups is wrorif-. The scene 
in figure 7-3 differs from that in figure 7-7 because there 
is not only symmetry in the location of groups of objects, 
there is also symirEtry 1n the placement of objects within the 
groups. This means that the scene is more syrmetr Ic to the 
machine in the sense that the symmetry detection program 
remains happy at a deeper level of inquiry. 

The machine knows LEFT*0F and RIGHT -OF are opposite* 
because Information ahouf. these pointers lies in the nenrral 
memory net. (ffnure 7^<?} Consequently, it is unnecessary to 
tell the program explicitly to substitute RTSHT-OF for LEFT- 
OF and vice-versa. One need only ask the symmetry pronran if 
there is symmetry with respect to oither the pointer LEFT-OF 
or fiIGKT-0F. The machine Itself can conjure up the 
appropriate substitutions by working through the OPPOSITE 
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pointer from whichever relation 1s supplied, be it LE.FT-0F or 

RIG I IT -OF. Similarly, if one asks for symmetry with respect 

to ABOVE* the program realizes that the proper substitutions 

are BELOW for ABOVE ana ABOVE for BELOW* IN-FRONT-OF gives 

BEHIND for IN-FROftT-OF and vice*versa for a somewhat unusual 

kind Of synrnetry question. 

Certafnlymlxturesarealsc re ssi bl e. One can ask for 

both left-right and above- below substitutions which is an 

abstraction of synmetry with respect to a point. 

Mathematically speaking, there is symiretry with 
respect to the particular point (0*0) if for every point 
fx*y) in the scene, there is also a point (-ju-v}. The 
way to check a drawing fs to imanine moving every point 
straight through the or i pin until it Is again the sa^e 
distance from the origin but lies in the opposite 
quadrant* For example, point P in the symmetric drawing 
in figure 7-10 goes to point P' . When the end result is 
the same as the original drawing, then the drawing is 
symmetr ic . 

More important is a combination of a I eft -right and an 

in-front-of -- behind switch* This one gives the mac h f ne a 

chance of realizing that two scenes are just front and hack 

views of the sane configuration as are the scenes In 

figure 7-11. 

Eventually I think the nachlne can come upon the 
symmetry notion in the same way 1t now learns about 
arches and houses* But at this point I do not think 
there 1s enough comparison describing capability. The 
needed step 1s the introduction of a program that 
generates global c-notes from the local ones already at 
hand, thereby introducing the kind of hierarchy into the 
comparison descriptions that is already the standard in 
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scene descriptions. One obvious ability of such a 
program would be that of noticlnn a preponderance of 
similar c-notes, I think that this and some of the 
double comparison ideas proven useful in ddng anaToay 
problems are just the things the machine needs to learn 
ahout symmetry. 



7.4 Elementary Identification 

Suppose a scene Is to be identified* 1f possible, as a 
HOUSE, PEDESTAL. TENT, or ARCH. The obvious procedure 1s to 
natch its description against these for each of the models 
and then somehow determine which of the four resulting 
difference descriptions Implies the best match. 

Recall that models generally certain nust-be satellites 
and must-not-ho satellites while ordinary descr ipt ions do 
not. Consequently, comparing an ordinary description against 
a model leads to a variety of c-notes not found when ordinary 
descriptions are compared, /'.monq these are nu st- he- satel 1 ite 
pairs, must-not-be-satell ite pairs, and various flavors of 
exits and supplementary-pointers. Such c-notes are decisive 
in the Identification process, 

■Consider the case where some pointer in a scene's 
description corresponds to its nust-not-be satellite in the 
model* This clearly means a relation is present that the 
model specifically forbids. The resulting must*nOt-be- 
satell ite-pair c-note In the difference network is such a 
serious association impediment that identification of the 
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unknown with the model is rejected outright, without further 
c on s iderat ton + This means that the near-arch in figure 7-12 
cannot be Identified as an arch because the network 
describing the near -arch has HARRYS pointers between the two 
supports while the model has r 'U ST -HOT-HARRY pointers in the 
same place* The combination produces a comparison 
description with a nust-not-be~sateli ite-pa ir c-n^te that 
positively prevents match. 

Identification with a particular model 1s also rejected 
If the difference description contains exits or 
supplementary-pointer c-notes which involve must-be 
satellites. Su:h u -notes occur when essential relations or 
properties are missing in the unknown . Thus the two bricks 
in figure 7-13 do not form a pedestal because the model for 
the pedestal has a HJ5T-DE -SUPPORTED -BY pointer wnere the 
scene of figure 7-13 has nothing. The result is a 
supplementa.ry~pc inter c-note involving the must*be satellite 
MUST -BE- SUPPORTED -BY, Again match fails. 

The case of a - kind -of-merce c-notes involves a slightly 
more complicated rule. Recall that ffieroe c-notes occur 
generally when two nodes share properties that in not 
identical* but which fall into the same general class* The 
situation must be one where two linked nodes exhibit closely 
related pointers to two other nodes from which paths of A» 
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K IKD -OF pointers lead to a common inter sect ion * Fiqurft 7-14 
shows such a situation* In this case the unknown is a kind 
of wedge while the corr esoond ing object in the node! must be 
,: k ] rs r! of brlsik, R C. t h l-.T H 1 T 3 •- J W.2\i\rP. r.i'idr, n r ot. ! ■■(: t -i , 

which directly leads to a merge c*note associate with a 
NLST-BE-A-KIND-OF pointer In the model. But the fact that 
the unknown has a property in the same class as a property 
required by the model is insufficient* To insure rejection 
of such matches^ the rule is: Refuse id ent if icat ion If the 
model's pointer contributing to the merfle c-nnte Is a must^ 
be^satell fte. 

Figure 7-15 summarizes the procedure used on each c- 
note* 

Match of the scene in figure 7-1 & against the PEDESTAL, 
the TENT, and the ^F!CK all lead to difference descriptions 
with c-notes that forbid identification. The PEDESTAL fails 
because a meroe indicates that the required A -KIND-OF 
relation between the top object and &F1 TC K is missing. The 
TENT similarly fails because both of its objects must be 
wedges, The ftRCK fails because the model has a MUST-BE- 
5UPPQHTED-BY pointer to an object missing in the unknown., 
This in turn causes a fatal ex it c-note in the difference 
d escr ipt ion, 

Of course the machine car also ratch the s amnio 
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pedestal, tent, and arch of figure 7-17 t figure ? - 1 B 1 and 
figure 7-19 against the. same list of models. It makes the 
correct identification, in each case.. 

The next problem emerges because some unknown nay 
acceptably match more than one model In a trial list, 
Suppose ore defines a new sort of arch that is just like the 
old arch except that the top object nust be a wedge rather 
than just any object* Call this new node! the WEDGE-ARCH. 
Then the scene in figure 7-20 certainly matches with both the 
ORDINARY-ARCH and the nen KEDGE-ARCH. There Is only one 
slight variation 1ft the difference desor ipt Ions. In the 
lOGE-ARCJI case, one has a must-ue-satel 1 tte-nafr c-note 
because the unknown has an A-KIND-OF pointer to WEDGE and the 
model has a MUST -BE -A -KIND -OF pointer. In the 01>» IHAR Y-ARC H 
case, there U simply an A-KIWD-OF pointer from the model to 
OBJECT, which with the unknown's A -KIND -OF pointer to WEDGE 
forms an a -k tnd -of-merge c-note. 

Of course there is nothing really wrong with reporting 
both ORDINARY-ARCH and UEDGE-ARCH as the Identification of 
the unknown. Still, given several possible Identifications, 
there- should be some way of ordering them such that one could 
be reported to be best in some sense. To do this I associate 
each kind of difference with a number and combine the numbers 
to form a score for each comparison* Figure 7-21 shows the 
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scale associat In q difference types with numbers. It evolves 

hour 1st leal 1 y from observations lite the following: 

1, The intersection c-note has an a shinned value of 1. 
This anchors the scale as all other numbers are fixed 
according to how <iood or bad the corresponding c-note 
seems relative to the intersection c-ncte, 

2* A mu st-be -sate! 1 1te-pa 1r c-note suggests good match 
even more strongly than the Intersection because T£ 
indicates that relations are present that are knoun 
to oe essential. A value of 3 flfyc-S- it three tines 
the weight of 9 s 1 m r ^ e- Intersection. 

3* Exit or supplementary -pointer c-nntes that Involve 
nu St - be -sat ell f tes are distinctively bad because 1 they 
indicate vital properties are m1ssinn H The value is 
a tiaiaginr -5* 

4. I'ther exit and sup? 1 ementary-poi nt er c -notes are ba cf 
but rat nearly so had, A ^core of -2 seems about 
right . 

5- '!u st-not-be -sa tell fte nafrs are very bad evidence 
indeed* The worst score of -6 is deserved. 

6* The a -kint-of -merge is nesitfve or negative depentflnn 
en whether either of the pointers are must-be 
Satellites, If a must -be satellite is invnlveri, an 
Important property is "■'ssinn, resulting in a -4, 
Ptherwfse, it indicates lof r se association, not as 
tight as that announced by an Inter sect i an 4 A .5 is 
used. 

Once differences are noted and number as soc latf ons 

are made, a program nust combine the numbers in a 

reasonable way* If SCORC[U:M] represents the score of 

comparing unknown u against model 'i, then i use 
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SCO*E[U:H] = H(l )fr(l ] + +U(n)N(n} 
wher e 

and 

■J(1 ) is the number associated v." f t h the 1th 
G" if fcrercCi 

W(i} fs the weighting factor that reduces the 
In f 1 uence of lower level differences* 

L(i) is the level of the 1th difference* 



Combining the terms additlvety is convenient, and the 
weighting terms, the '.-'s, handily reduce the influence of 
the lower level d iff erenc es, I have no stronger reasons 
for usinn this linear formula, and ft is hot tcnetliinn to 
be defended to the death. But I do net think it would pay 
to put much effort Into tuning such a f omul a because unre 
knowledcc About the priorities 0* differences should lead 
to far better programs that do not use numbers at all, 
7,5 Identification In Context 

Ex ah In. e f inure 7-22. Mot fee that object 6 seens to 
be a brick while object D seerns to be a wed as. This is 
curious because D and show exactly the same arrangement 
of 1 f n e s and faces, The result also seems at odd s with 
the machine's node Is and identification process, as 
described sc far t because so far anythffiq identified as a 
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wedge must have a triangular face» 

But of course context is the ftxclanat ion . Different 
rules must be used when programs try to identify objects 
or groups, of objects that are only parts of scenes rather 
than the whole scene. In the case where the question Is 
in he :. h er c r not t he 14 hoi e sc c n e car be f d en t f f ' 1 ed as ( 
particular rniodeli it is reasonable to Insist that all 
relations deemed essential by the model be present^ while 
all those forbidden* he ahs^nt. But when the question Is 
whether Of not a few parts of a scene can bo identified as 
a particular model*, then there is the possibility that 
some frtportant part maj be obscured by other objects. In 
these situations, my identlf 1ca t inn program uses tw) 
special heuristics: 

First, the coincidence of objects lyinq in a line 
seems to suggest that each object is the same type as the 
one obscuring it unless there fs nood reason to reject 
this hypothesis* This is what suggests object D is a 
ned a e in figure 1-77. 

Second, essential preoortics in the model may he 
absent in the unknown because th« parts Involved are 
hidden. This 1s nhy identification of object I) with werfne 
works even though Q lacks the otherwise essential 
trianoular face. The reou irement that forbidden 
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properties dc r<0t occur remains in force, however. 

Elaborate wort can be done on the problem of deciding 
if the omission of a particular feature of some model >s 
admissable in any particular situation, ^y nrOfjrar- takes 
a singularly cruac view and Ignores all omissions. 
Reject fon of the hypothesis that the obscured 1s like the 
obscurer happens only if the machine notices details 
specifically forbidden by relations 1n the model. Thus 
the effort is not to select' the test matching model, but 
only to verify that * particular identification is not 
contradictory. This reans that object Li in finure 7-21 
1s confirmed to be brick-like while br1ck-ness 1s denied 
to because of the ruinous apparent triangularity of the 
side face. 

Of course if the propagation of a property like 
brick-ness or wDdqe-ness d pvjn a series of objects is 
interrupted^ tben the unknoun nu St be conpered with a 
battery of models, ulth the program still forolving 
omissions* but now searching for the best of many possible 
Identifications. 

Ho matter what the method by which a partly obscured 
ob;ect is identified, the use of a PPOGABLY-A-KIND-OF 
pointer instead of the basic A- KING "OF is used to qualify 
the conjectured relationship between the object and the 
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nod el it f& Id er>t if teri with* 

Ffoure 7-24 shows bow the various pieces of this 
procedure fit together , and f inure 7-£& shows what happens 
when ft moves clown a simple row of objects, 
7.6 Learning from Mistakes 

Suppose the prop/rani attempts to identify the scene in 
figure 7-2G as a pedestal. identification fails because 
the resultfnp type of a-Mnd -nf-merge c-note cannot be 
tolerated. Still ft would be a pity to throw away the 
information about why the natch failed* Instead the 
:• :.!i..rv- n? >■■■■■ v-.ci ".■t'-i'-' a "or*, car 1:h u s cH -c <tu n-v^: k.- 
new identification candidates. 

The way this works is quite simple. First the 
machine spends idle tine con par inn the various tnodels in 
ft a arrcamentorfiim with each other, whenever the number of 
differences observed art few fc a simplified description of 
those differences is stored. Thus the machine knows that 
a house Is similar to a pedestal, from which it differs 
only in the nature of the tnp object. 

These descriptions link the known models together in 
a sort of similarity network* {figure 7-27) 

This network and the difference deser ipt i ons noted 1n 
the course of identification failure help decide what 
model should be tried next. The description describing 
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the d iff erences between an unknown and a particular model 
is compared *1th the descriptions of the similarity net. 
If the difference between the unknown a r>4 a particular 

model matches the difference between that model and some 

other models then identification with that other model is 

\ i kel y, 

For example, the scent- cf figure 7-2 6 relates to the 
model of a pedestal in roughly the same way that the model 
of a house relates to the model of a pedestal. House is 
consequently elevated to the top of the list of trial 
models- Figure 7-2G clarifies the procedure. 
7.6*1 Similar ity [rescr lot ion s 

The similarity descriptions are simplifications of 
the comparison descriptions and are part of the 
description of each pointer that relates similar concepts. 
'«lnen a losing identification reminds too program of some 
difference structure it has seen, no serious cpmn ftment Is 
made and mistaken conjecture* do not hurt much. 
Consequently it is desirable to strip the difference 
descriptions to the Important elements, thereby saving 
storage space and increasing matching speed,, even though 
some wrong models may be proposed as 1 Hely 
identifications* The simplified description therefore 
consists in part of a sort of skeleton, [figure 7-29} 
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Intersection c -notes and others associated with positive 
numbers on the evaluation scale are ignored because only 
the disruptive c-notes are of interest here* These 
disruptive c-notes, which sup/nest poor match, are huna on 
the skeleton. 

Figure 7 -3 D shows the simplified difference 
description resulting front comparison of the house model 
with the pedestal model. Notice that it 1s exactly the 
same under this simplification transformation as that 
resulting from comparison of the pedestal model and the 
scene in figure 7-26. 
7.6.2 definition of quantitatively Snail 

This whole similarity scheme depends on the fact that 
two models may have only one or a few differences that 
make them strongly d ifferent in a qualitative sense* 
Irdmv,, the siiri "arity 1 irks si-cu '.-:i ex int. m;lj- ;.y-; ihn 
models Involved are reasonably close in the sense of 
producing f ew d iff eronces* When this i& true, an unknown 
that nearly Identifies with onemndel in the sense of feu 
differences is assured of ma tc hi ncr well with the other ► 
particularly when the two sets of difference? natch, Thus 
there must be some rule fcr dec id inc if Xht> number of 
differences Is sufficiently small to warrant a pair of 
pointers in the similarity network. Currently thp machine 
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con sfd or & suff ic lentT y small to mean the number of 

:n "i Sna tc h-cau sine? c-notes is either tcss than trn or leis 

than one-third of the number of other c-notes. 

7.7 The Needle in the Haystack 

The scene of figure 7-3 1 is curious in that ore can 

find an arch, a pedestal , a house, and a tent In It ff one 

is looking for them, Dut if the.y are not spec i fie* 11 y 

searched for, mention of these particular models is 

unlikely to appear In a description of the- scena. 

Although the corf iterations are present, they are hidden 

by extraneous objects sn veil that pen oral grouping 

programs are unlikely to sort the pi out. Yet the Question, 

"Does a certain model appear in the scene?" is certainly a 

reasonable one K One way to attack it dfylrtes nicely Into 

three partss 

K Find those objects In Uif* scene that have the best 
chance of being, identified with the node 1 . If the 
model has unusual pointers or references unusual 
concepts* the program pays particular attention to 
them. Similarly,, extra attention is paid to the 
emphasized parts of the model, for if mates cannot 
be established for them^ solid identification 
cannot be affirmed. Happily, Py standard network 
hatching program does these things without 
au gnenta Hon , The result is a set of links 
between the objects of the model and their nearest 
an-alogues in the scene. The other parts of the 
scene remain unlinked and end up appearing in exit 
c-no t es t 

2 t Once a good group of objects is picked, then the 
pointers relating these objects to the other 
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objects in the scene aire temporarily forgotten* 
In human term% this fs like painting the subgrnuD 
a special color or otherwise reducing possible 
confusion from relations with the surroynd 1na 
objec ts, 

3* Finally, with the best group of objects set Into 
relief by the previous excision, the ordinary 
ident if icaton routines are applied with the 
expectation of reasonable performance* 

The folly of direct application of the Identification 

programs lies In the iryrlad irrelevant exit c*notes that 

the extra objects in the scene timid cause. Such clutter 

leaves the machine as bewildered as ft does humans. 

7*8 Reacting to Identification 

Once Identification of a substructure happens^ the 
discovery should contribute to the store o* Stnowledge, 
Floure 7-32 Illustrates some of the more obvious things 
done after ident ff tea tion of the house and arch fn 
figure 7-33, The highest level node nn longer connects 
directly to the individual objects. Instead* those 
objects dangle from new sub scene nodes by ONE- PART- IS 
pointers. Similarly the old top level concept points to 
the new sub scenes with {!f;t-P/.RT- IS. The sub scene nodes 
naturally point by A- KINO -OF to the models they identify 
with. 

Rather more can tie d«ne if the irachlne knows 
something about how the relations of a group's components 
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to eKtcrr.il objects dictate the relations of the orcup. 

Any knowledgeable machine knon-s that a house conf inurat 1on 

rests on whatever its bottom object rests on* More 

generally, the following rule seems reasonable: 

Sypocse A and B are groups of objects identified as 
substructures, then if an object of A relates to an 

object of & by SU PROP TED -BY , then substructure A 
relates to substructure & by SUPPORT EO-Dy. 

In can sentence, the net in figure 7-32 becomes that In 
figure 7-34, 
7*8.1 Examples 

Using this same procedure, the scene fn Moure 7-35 
soon reaches the state of f 11 un tnat "Tor, sh^wn fin 
figure 7-37* By examining either the picture or the net, 
it is Easy to see that the arches AT, AL* and A*? 
themselves constitute a sort of super-arch with arches as 
parts instead of objects. The machine foes not refuse 
this substitution since the model for ARCH has only ONE- 
PART- IS pointers to &RICK and OBJECT, not OHE-PART-MUST-BE 
pointers. The matching score is slnply lower than it 
would be for arches made of bricks. The final description 
essentially states that the scene consists of a sort of 
arch supported by an arch composed of three arches, 

Figure 7-36 shows a richer example Including 
instances of a pedestal. Again the machine Identifies 
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groups, establishes relations between the Identified 
groups, and then tries to identify groups of groups, 
reporting eventual ly that there is 4ft jreh composed of an 
arch on top with two pedestals for supports. It then 
notices that thts general ized arch is supported by two 
ordinary arches. But the generalized arch on too of two 
Supporting arches ana In is a kind of arch, the fifth and 
last discovered^ 
1/6,2 Other Relations 

I have not thought much about the calculation of 
other group properties. It seen-s reasonable, however, 
that a set of programs using the following ideas should 
work to some extent, albeit crudely. To find Jt| -FRONT -OF 
relations between groups one can use the above rule for 
SUPPORT with the obvious exchange of IN-FROUT-QF for 
SUPPORTED -BY. To establish the size of a qroup one can 
add together the fndlvlrtual areas of its objects. To 
check for LEFT-OF and PIGIIT-OF, one uses the center of 
area tnd extreme points of the entire group rather than 
those of an Individual object, But otherwise the left- 
right algorithm may remain the sane. 
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3 Closing Remarks 

a J A System 

The flow diagram in figure 3-1 sho^s how the 
techniques fit together with those of others to form a 
primitive scene-perceiving system. At the very beginning 
lies the scene, from which all infornation ultinatoly 
derives, A program developed by Griffith [7] watches the 
scene through an eye resembling television camera* The 
result is a line drawing. Next programs of Mahabala [1] 
and Guzraan [2] classify vortexes and group regions into 
bodies. Next ts a stage in which object identification 
1s done. Following closely, one has the determination of 
object-object relations and then group identification. 
Finally there is identification of group-group relations, 

Dcyond this* action depends on intent* On one path 
one finds attempted Identification of the entire scene 
with a known lodel or models. Cn another an effort is 
made to find an instance of some particular model in the 
scene* Still another rath involves use of the 
description to help form new concepts. 
.2 Cone lu sions 

This collection of ideas and techniques supports four 
major contentions, each of which depends on those 
preceding it. These things are small steps for a wan , but 



CONTRIBUTE TO FORMATION 
OF A KEtf CONCEPT 



4 



FIHE INSTANCE OP 
SOtfE ttlDEL 



IDENTIFY SCENE 



find grouf-gequf 
relictions 



f 



ides/hpy groups 



z: 



FISD OBJECT-OBJECT 
RELATIONS 



I 



HiEHEIFY OBJECTS 



USE GUZMAN'S 

REGION GROOVING 

PROGRAM 



1~Z 

—h _ 



USE MALIABAiJ^S 
VERTEX CLASSIFYING 
PROCRAH 



USE BIRFORD'S OR 
GRIFFITH'S LINE 
FIMHU& PROGRAM 



U&E EVE 



FIGURE &-1 



Z49 



q1fl.nt leaps for a machine* 

1. fl computer can produce a detailed scene 

description consfstino of the same sort of facts 
humans observe, 

2« These descriptions lead in turn to descritu 1 on s of 
how scene* compare with on* another. 

3, An understand fng of hew scenes compare permits the 
computer to tearn models for new c one ept s from 
examples and leads to a new way of thin fc In a about 

1 earn intj, 

4. These models finally endow the machine with the 
ability to recorinize instances of previously 

1 earned co ncepts + 

ft. 3 Background Issues 

In a more cosnlc sense, the goat behind this work f$ 
to make a machine that can understand the environment just 
as we humans seem to. Some critics of Artificial 
Intelligence think that this is not possible, perhaps 
because they cannot Imagine how 1t can be done. 1 think 
the real hang up must He tn the understand 1ng one has 
about the notion of u nderstanrf 1no. fi review of a few 
dictionaries convinces me that the editors arc hard 
pressed to define the word without uslnq it. It is as if 
it taere a word so basic that it cannot be described in 
s imnl er terms t 

Cut surely to understand nu st involve the formation 
of a descriptive plateau of knowledge l.ving Somewhere 
between raw, totally unprocessed data and detailed answers 
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to problems, I do not wish to belabor this point, but I 

feel that the sort of abstraction represented by the 

network description of a scene can be viewed as 

con st itu~ inrj a scrl. r: f understanding. If so, depth of 

understanding corresponds, roughly to the elaborateness of 

a description, dense networks suggesting more 

under stand ing thai sparse dries. 

Another notion of long standing concern to phUosuohy 
is that of the ideal form, Vet Tittle wort seems to have 
gone intc careful study of what humans mean hy such simple 
concepts as that of the TABLE, I believe study and 
improvement of thd concent generator constitutes a fresh 
approach to this problem and may lead to interesting new 
resul t s. 
6.4 Suggestions for Further V'ork 

Improvements to and extensions of this work can be 
understood in terns of two extremes: minor change and 
major overhaul, The minor-change category Is large 
L'ec<iu?,n :-f! Mr host -r^ri'.v -7 na 1 in sui:r- unrk rust bn ?. 
complete system. A complete system, bowr-ver flimsy, 
serve; to guide resource allocation into the most deserved 
problem areas. Without experience with such a system, One 
risks suffering from the phenomenon of diminishing 
returns, expending great effort for marginal improvements 
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on relatively strong pieces rjf t r=e syster. Put the 
natural result is that there is much room for further 
-improvement of the system's parts* Some possibilities 
have already been mentioned, but it fs appropriate to 
Dent ion then lie re as a convenience for those who wish to 
work In the area* 

1, Nearly all the programs that establish relations 
be twee i) objects can be Imc-roved* The' program that 
looks *or support blunders sometimes because 
obvious bottom lines are over looked anc sometimes 
because a scene has tipped hac ground objects. 

The program that looks for 1n-ffont*Of 
relation ships cannot handle situat'C"? i'- ■ .' h -' c i ■ 
objects are aligned, *iany of these programs coulri 
benefit from a program that could imagine hidden 
1 ines* 

2. No distinctions are inado between ofcvlcus, 
unarguable properties and borderline cases. It 
n icrht be pood if the Analytic prnnrars could 
report things 1 He certs inly-1 eft -of or sort -Of- 
1 eft -of instead of invariable, undifferentiated 
left-of . 

3* The rules for the Identification of a scene with a 
model need refinement, The weighting associated 

with the varices differences enf! the way those 
weights arc cr-i l : -m\\ ]■■■<•>?■ - specious Quality* It. 
T^ould be fine 1f some way could be devised to 
eliminate the numbers a It cnether* perhaos throucih 
a more intelligent prouratn with a built-in 
understanding of priorities. 

4. The scheres for ree ocm 1 zing reasonable clusters of 
objects is particularly primitive and has 

undergone too little testing)* Mechanisms oust be 
found for producing and handling alternatives to 
the first partition devised. 

5* The entire concept generation procedure and its 
ramifications certainly should absorb great 
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at tent inn, More powerful net iiods fnr recognizing 
important differences are needed, The (tacrine 
should have a fatuity by which it can complain or 
ask questions if the teacher is doino poorly, 
Generalization to functional properties fs a must. 

6. The network matching proqran, althnyah 1t Is at 
the core of the en t fro system^ Is a hastily 
programmed, slow and stubborn itu-nhl et»un. An 
Improved version uould simultaneously increase the 
power of the many system activities that use it. 

The other kind of Improvement is the major overhaul* 
This 1s not aincd at sophist icat ine an existing, part of 
the lyston, but rather focuses on the spectacular 
Increases in power derived fron bolder. Mgner Ideas. I 
discuss two sue h possibilities below k Re'ireta LI y> these 
problems ars difficult to formulate and del tin it. 

First fs the general problem of introduc fnci bi- 
directional in format Ion flow. Glance again at the flow 
diagram Of figure 6-1 . "Is lice that al" c r the arrows 
point fron bottom to tor, indicating that all information 
moves in one direction only. There is as yet no way a 
process can dfseourse with and modify the behavior of any 
process a t tine tela v.' it. 

It seems clear that for high level command to affect 
1 ok level operation in a non -trivial v,ay, there must bo 
some alternative or additional method that can be thrown 
into tattle. As yet few such points of influence exist fn 
my systeru There are two left-of ■»- r1njht-of procedures,, 
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and there is the- option of using or not usfnp various 
relation -find Ing and grouping procedures. Cut the forte 
of an internally Interactive system will probably involve 
selection of one method from several p-os si b1 lit ie 5 amonn 
which there are trades between speed and accuracy. 

Arc t her amorphous problem is that of wedding the 
visual capabilities of this system with other systems that 
specialize In different kinds of perception. A real robot 
should understand the environment not only in te<-irs cf 
vision but also in terms of touch, sound, language, and 
perhaps other mediums, U ndeTStand inj each of these Is a 
major problenu but as work proceeds, there hill be the 
su per -pr obi em of understand ing ho^ various perceptions of 
the environment should interact to form a unified 
under stand ino,. 

I understand neuroanaton ical evidence is that 
evolution has Come to arins with this problem only 
Jately and only in nan with z.ny finesse. Norman 
Geschwind reports that monkeys havp very Tint ted 
ability to correlate things they learn via the 
visual, auditory, and sonesthetic senses [a]. Indeed 

. may be reasonable to say each monkey is really 
three monkeys occupying the same skull, a visually 
oriented monkey, a aud Hory one 4 and a somesthetie 
one, Man apparently avoids this through a chunk of 



cortex that soroehou watches up these perceptions. 
The chunk believed responsible by Geschwind is cal 



the inferior parietal lobule, 
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Append ix 

This appendix is a cursory introduction to the 
network matching program essential to many of the system's 
operations. Its job is to determine which nodes of two 
descriptions best correspond. The corresponding; nodes are 
said to be the linked pairs. 

The program starts trUh the entry nodes of two 
de Str 1pt iCPS* It immediately inquires if there is 
evidence that the two nodes should be considered a linked 
pair. The answer is j/ps if both nodes have a common 
pointer to some common intersection node. Thus X and X ' 
sn a linked pair in figure A- I, but they are certainly 
-. Lit in figure fl-2, :irct: nrdss X ?. nd ?! ' -ni ■; c neither 
pointers nor nodes in comnon t 

If no linking can occur, the program moves donn one 
level through the most common pointer present to dauqhters 
of the currently Inspected nodes and tries to find linked 
pairs among them* In figure A-3, the entry nodes are not 
linked on first inspection since there are no common 
pointers to any conirron rode. They both have daughter 
nodes, however t and these are nest examined. P is the 
most numerous pointer, so ncde; C^ t C2, CI', and Z2* form 
two groups in which the program tries to find rrood pairs 
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to establish as linked. (figure A-4) Since Cl and CI 1 hoth 
have the sane pointer to a comnon node they are good 
candidates for the formation of a 1 in ked pa ir . Nodes C 2 
and C2' are even more alike, however, since they have two 
pointers to two common nodes. C on se ouontl y, C2 and C 2 ' 
are linked first, with action on CI and CI 1 postponed. 

Each time a pair of nodes is linked^ the program trys 
to link up any of their daughter nodes that are not 
intersections, In the example this mo^ns that an effort 
is made to find a linked pair between the left node, El, 
and the right nodes, El 1 and E2 1 , all of which are found 
at the «nd of ? pointers-, (figure A-5) This is the first 
example of a contcst K Both [1* and CE ' share Connor 
intersections wi th CI. The winning pair picked by the 
machine 1s always the one »fith the highest count of conn on 
pointers to Inter sec ton & or previously linked pairs. In 
this case t El -El" scores hioher because there is net only 
a set of pointers to the Intersection node* but also a set 
to the previously lfnked pair C2-C2 1 , 

The linking of El and El 1 causes examination of their 
daughters^ Fl on the left> Fl r and F2 1 on the right. 
(figure A-6 ) Fl and F2 ' both have a pointer R to a common 
node, Fl \ however » has the nust-he version of the 
pointer P. to the same node* Such satellites occur 
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frequently in models, indicating mandatory relat Ions* 
Priority Is given to matches in which satellites 
correspond to the pointers they are modifications of. 
Thf$ means the R with MUST-BE-P pointer match between 
nodes Fl and M 1 has somewhat nore welnht than the match 
of R with R that one would have between Fl and F2*. The 
Fl-Fl' match therefore is considered the better one and 
results in a linked pair* 

Fl a.nd Fl' have no daughters to he paired so no 
further penetration of the 'net occurs. The next job fs to 
re-examine the next higher parent node's because links just 
formed may provide enough nnw evidence to link tiw higher 
nodes* In this case backup first considers nodes El and 
El \ El and £1' are already linked, Attention therefore 
pops up st 11 1 another level toCl and CI*, CI and Cl k are 
linkcc b 3c -exam inat ion of thefr unlinkecf daughters, E> and 
E2, reveals nothing new. 

Once more the programs, attention shifts upward,, this 
tine to A and A', f lon thorc- is A pair of pointers tn 
1 fnked nodes supplying linking evidence (figure A -7} A 1s 
consequently linked to f*.\ 

Next comes further examination of the remaining 
daughters of A and A'* C] 1 s now the most common pointer 
to unaccounted for nodes, pointing as It does to Gl, (12, 
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and G2". (figure A -3) Aneng these nodes there Is the same 
anion nt of evidence for I In kin? Til and G2' as there Is for 
linkinn t£ and G2\ When this is the case and no further 
evidence can be collected from linking lo^er level nodes^ 
a linking 1s selected randomly from those possible. Here 
G2 and S2 ' are linked, 

This leaves only reesun fnat ion of C£ and C2 1 . The 
intersection evidence is clenr ard they are linked. Since 
A and A h have no more daughters and since they are the 
entry nodes, the process terminates reporting the linkages 
indicated on the fully displayed network of figure A-9. 
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